# CodeNeRF: Disentangled Neural Radiance Fields for Object Categories

Wonbong Jang Lourdes Agapito
Department of Computer Science
University College London
<EMAIL_ADDRESS>
###### Abstract

CodeNeRF is an implicit 3D neural representation that learns the variation of
object shapes and textures across a category and can be trained, from a set of
posed images, to synthesize novel views of unseen objects. Unlike the original
NeRF, which is scene specific, CodeNeRF learns to disentangle shape and
texture by learning separate embeddings. At test time, given a single unposed
image of an unseen object, CodeNeRF jointly estimates camera viewpoint, and
shape and appearance codes via optimization. Unseen objects can be
reconstructed from a single image, and then rendered from new viewpoints or
their shape and texture edited by varying the latent codes. We conduct
experiments on the SRN benchmark, which show that CodeNeRF generalises well to
unseen objects and achieves on-par performance with methods that require known
camera pose at test time. Our results on real-world images demonstrate that
CodeNeRF can bridge the sim-to-real gap. Project page:
https://github.com/wayne1123/code-nerf

Figure 1: CodeNeRF represents variations of shape and texture across an object
class. Left: At test time, given a single input image, the trained model can
be used to jointly optimize camera viewpoint and shape/texture latent codes.
Right: Object shapes, textures and viewpoints can later be edited simply by
varying the latent codes, offering full control over synthesis.

##  1 Introduction

Synthesizing novel views of unseen objects given a sparse set of input views
or even a single image is a long-standing problem in the fields of computer
vision and graphics. Synthesis methods require both an accurate representation
of the 3D geometry and appearance of objects, and the ability to offer control
over changes in viewpoint, shape or texture to render different objects of the
same category.

Traditionally, discrete scene representations have been used, such as meshes
or voxel grids that store geometry and appearance information either
explicitly or via learnt neural features [23]. However, their discrete nature
limits their representation power and resolution. With the recent introduction
of scene representation networks (SRN), Sitzmann _et al_. [24] propose to
learn a continuous function that map 3D locations to features of scene
properties. Crucially, SRN does not require 3D supervision and can be trained
end-to-end with a differentiable ray marcher that renders the feature-based
representation into a set of 2D images. While SRN allows generalization to
unseen objects, accurate camera pose estimates are required at inference time,
and shape and appearance are represented in an entangled manner.

Mildenhall _et al_. [15] extended the neural representation to store volume
density and view-dependent radiance values, enabling highly photo-realistic
new view synthesis of complex real-world scenes that capture viewpoint
dependent effects. New view synthesis is done by querying the network at each
specific spatial location and viewing direction, followed by classic volume
rendering to output image pixel intensities. While the quality of synthesized
images is impressive, it requires a large number of images, and must be
optimized for each new scene independently.

Figure 2: CodeNeRF disentangles geometry, appearance and viewpoint by learning separate embeddings for shape and texture and a fully connected network F($\theta$) that maps 3D locations and ray directions to density and RGB values. |  | Deep
---
SDF
SRN | NeRF |  | Pixel
---
NeRF
CodeNeRF
Learned Prior | ✓ | ✓ | ✗ | ✓ | ✓
Supervision | 3D | 2D | 2D | 2D | 2D
Encoder | ✗ | ✗ | ✗ | ✓ | ✗
Pose Optimization | - | ✗ | ✗ | ✗ | ✓
| Disentangled
---
Shape & Texture
✗ | ✗ | ✗ | ✗ | ✓
Table 1: Related prior work on neural scene representations.

Neural representations have also been used to learn deformation priors that
encode the variation of object shapes across semantic categories using direct
3D supervision [7, 2, 17, 14]. DeepSDF [17] is an auto-decoder architecture
that jointly learns a latent embedding and the weights of a fully connected
network that maps 3D coordinates to signed distance values. This
representation uses test time optimization to estimate the shape code
associated with a new unseen object. While DeepSDF [17] has been an extremely
popular representation, successfully used as a shape prior to drive 3D
reconstruction of object categories from multiple images [21, 20], its
training requires 3D supervision and cannot be learnt from 2D images only.

Figure 3: We illustrate the ability of CodeNeRF to disentangle geometry and
appearance by synthesizing new shapes or textures via latent interpolation. We
show two reference images (left and right). CodeNeRF first estimates the
camera pose and shape+texture codes for these images and then interpolates
between them.

Contributions: We present CodeNeRF, a novel 3D-aware representation for object
categories that learns to disentangle shape and texture. During training,
CodeNeRF takes a set of posed input images and simultaneously learns different
latent embeddings for shape and texture, and the weights of a multi-layer
perceptron to predict volume density and view-dependent radiance for each 3D
point by enforcing multi-view photometric consistency. At inference, given a
single unposed reference image of an unseen object, CodeNeRF optimizes shape
and texture codes as well as camera pose. Our disentangled representation
provides full control over the synthesis task, enabling explicit editing of
object shape and texture simply by modifying the respective latent codes (see
Fig 1). We demonstrate single view reconstruction, novel view synthesis and
shape/texture editing on the SRN benchmark and real-world images.

While our work takes inspiration from other continuous neural scene
representations [15, 24], it addresses many of their limitations. Unlike NeRF
[15], CodeNeRF is not scene specific and can model the variations of shape and
appearance across an object class. In contrast to SRN [24], CodeNeRF
disentangles geometry and appearance offering explicit control over shape and
texture for synthesis tasks. Unlike both, CodeNeRF does not require knowledge
of camera pose at test time, estimating it via optimization. Inspired by
DeepSDF [17], we adopt an auto-decoder architecture but depart significantly
as we only require 2D supervision and can disentangle shape and texture
variations across an object category. See Fig 2 for an overview of CodeNeRF.

##  2 Related Work

We focus our literature review on learning based approaches to 3D
reconstruction and new view synthesis.

3D Reconstruction from 3D supervision Voxel-based methods were amongst the
first [34, 3, 32] to propose learned representations for 3D reconstruction
with 3D supervision. However, volumetric representations are fundamentally
limited in terms of representation power and resolution and fail to capture
surface details. Although point clouds are a common alternative to voxels,
their unstructured nature and permutation-invariance make them difficult to
handle using cnn architectures. PointNet [18] offered a unified architecture
able to deal with unordered point sets via maxpooling, that may be used for
multiple downstream learning tasks. Mesh-based representations have also been
used for 3D shape estimation given 3D supervision [7] or applying graph
convolutions [29, 6]. However, these methods still fail to capture fine
details. DeepSDF [17] pioneered a new popular direction to represent 3D
geometry via a continuous mapping from 3D points to an implicit representation
of shape. DeepSDF jointly learns the weights of a fully connected network that
maps 3D coordinates to signed distance values, and a latent embedding. While
in principle DeepSDF can represent shapes at arbitrary resolutions without
increasing its memory footprint, its training requires 3D supervision. Other
concurrent approaches used continuous implicit occupancy networks [14, 2] but
also require 3D annotations. A CNN-based encoder to extract local features to
provide conditioning priors was used in [35]. However, all these methods
require heavy pre-processing of the training data to compute the SDF of query
points. In addition, if a mesh is required during optimization for rendering,
marching cubes must be used, which is not easily differentiable [11]. Mesh-
based, SDF or occupancy representations pre-learnt with 3D supervision [14, 2,
35] can be combined with a CNN encoder to perform single view reconstruction.
However, as noted in [27], these models often perform recognition instead of
3D reconstruction.

3D Reconstruction from 2D supervision The emergence of differentiable
renderers [8, 10, 19] allowed to obtain gradients through the rendering
process, enabling them to be embedded into end-to-end learning frameworks.
DeepSDF was used as a deep pre-learnt shape prior for 3D shape optimization
from multiple images [21] requiring 2D supervision only. MeshSDF [20]
circumvents the non-differentiability of marching cubes and uses 2D
supervision based on a silhouette loss. A mesh-based representation is
optimized from 2D images in [12]. The disadvantage of many of these methods is
that the learning of shape priors is decoupled from the shape estimation.

Neural 3D representations DVR [16] proposes a differentiable rendering
formulation for implicit shape and texture representations that can be learnt
from multi-view images and object masks only, without the need for 3D
supervision. Sitzmann _et al_. [24] introduced SRN (Scene Representation
Networks) a 3D-aware representation that learns a scene prior via a continuous
mapping of 3D spatial locations to features of scene properties. Similar to
our approach, once trained, the model can be used as a learned scene prior via
test-time optimization. However, unlike CodeNeRF, test-time optimization for
SRN requires known absolute camera poses, which can be seen as strong
supervision.

Neural Radiance Fields (NeRF) for new view synthesis NeRF [15] extended the
neural representation to allow to capture viewpoint varying image properties
by storing volume density and view-dependent radiance values which enabled
photorealistic new view synthesis of complex real-world scenes. Many new
extensions to NeRF have been recently proposed, some concurrent to ours. GRAF
[22] introduces the idea of _conditional_ radiance fields into the generative
adversarial network framework, to convert it into a 3D aware generative model.
In addition to new viewpoint synthesis, similarly to ours, their approach
allows to modify shape and appearance of the generated objects via shape and
texture latent embeddings. However, unlike ours, their method is a purely
generative model, trained on an adversarial loss over patches. As such, it
cannot estimate shape, texture or camera pose given an input image so cannot
be used for single view reconstruction. PixelNeRF [37] can generalise to
multiple scenes by using an image encoder to condition the neural radiance
field on image features and does not require test-time optimization. However,
unlike our approach, there is no disentanglement of geometry and appearance
and therefore no control over shape or texture editing. iNeRF [36] uses a pre-
trained NeRF to optimize the camera pose while NeRF\--[31] jointly estimates
the neural representation and camera intrinsic and extrinsic parameters but
both are scene specific.

##  3 Methodology

Figure 4: CodeNeRF architecture. $\gamma_{x}(\textbf{x})$ and
$\gamma_{d}(\textbf{d})$ are positional encodings for 3D point coordinates and
viewing directions respectively. Volume density $\sigma$ does not depend on
texture codes $\mathbf{z_{t}}$

Given a training set of N images depicting M objects across a semantic class,
along with their respective camera intrinsics and pose parameters
$\mathcal{V}=\\{\mathcal{I}_{i},\text{K}_{i},\text{T}_{i}\\}_{i=1}^{\text{N}}$,
CodeNeRF jointly learns the weights of a multi-layer perceptron
$\text{F}_{\Theta}$ that encapsulates the geometry and appearance across the
observed objects, and separate latent embeddings
$\\{\mathbf{z_{s}}^{j},\mathbf{z_{t}}^{j}\\}_{j=1}^{\text{M}}\in\mathbb{R}^{256}$
that disentangle shape and appearance respectively. $\text{F}_{\Theta}$ is a
disentangled neural radiance field that takes shape and texture codes as
inputs and maps scene coordinates $\mathbf{x}$ and viewing directions
$\mathbf{d}$ to their corresponding volume density $\sigma$ and RGB colour
values $\mathbf{c}$:

$\text{F}_{\Theta}:(\gamma_{x}(\mathbf{x}),\gamma_{d}(\mathbf{d}),\mathbf{z_{s}},\mathbf{z_{t}})\xrightarrow[]{}(\sigma,\mathbf{c})$
(1)

Following [15] we use positional encoding $\gamma(\cdot)$ for both scene
coordinates $\mathbf{x}$ and viewing directions $\mathbf{d}$ to capture high
frequency details. We employ $10$ frequencies for $\mathbf{x}$ and $4$ for
$\mathbf{d}$. We design the architecture of the decoder to take advantage of
the fact that the volume density $\sigma$ depends only on the 3D point
$\mathbf{x}$ and shape code $\mathbf{z_{s}}$, while the RGB color depends in
addition on the viewing direction $\mathbf{x}$ and texture code
$\mathbf{z_{t}}$. The first layers of the MLP $\text{F}^{{s}}_{\Theta}$ map
the input 3D coordinate $\gamma(\mathbf{x})$ and shape code $\mathbf{z_{s}}$
to the volume density $\sigma$ and an intermediate feature vector
$\mathbf{v}\in\mathbb{R}^{256}$. The second part of the network
$\text{F}^{{t}}_{\Theta}$ takes $\mathbf{v}$ and $\mathbf{z_{t}}$ as input and
outputs the RGB colour as shown in Fig. 4.

$\begin{split}\text{F}^{{s}}_{\Theta_{s}}&:(\gamma_{\text{x}}(\mathbf{x}),\mathbf{z_{s}})\xrightarrow[]{}(\sigma,\mathbf{v})\\\
\text{F}^{{t}}_{\Theta_{t}}&:(\mathbf{v},\gamma_{\text{d}}(\mathbf{d}),\mathbf{z_{t}})\xrightarrow[]{}(\mathbf{c})\\\
\text{F}_{\Theta}&:\text{F}^{{s}}_{\Theta_{s}}\circ\text{F}^{{t}}_{\Theta_{t}}\end{split}$
(2)

Conditioning on Learned Shape and Texture Embeddings Inspired by SRN [24] and
DeepSDF [17], CodeNeRF adopts an auto-decoder architecture with decoupled
shape and texture embedding spaces. During training, embedding vectors
$\\{\mathbf{z_{s}}^{j},\mathbf{z_{t}}^{j}\\}_{j=1}^{\text{M}}$ are optimized
jointly with the parameters of the network $\text{F}_{\Theta}$. Unlike DeepSDF
[17] the shape and texture embeddings can be trained end-to-end from images
only, without requiring any 3D supervision. In contrast to SRNs [24] shape and
texture embeddings are decoupled and can be used as separate controls for
editing/synthesis. Figs. 1, 3 and 9 show how the embeddings can be used as
conditioning to render smooth interpolations between different shapes/textures
in the latent spaces. Jointly, the shape/texture embeddings and the decoder
network $\text{F}_{\theta}$ act as learned priors that model shape
deformations and texture variations across a semantic category. Once the model
is trained, given a single input image, it can be used for one-shot
reconstruction of an unseen object, via a test-time optimization that
estimates the corresponding shape and texture codes
$\\{\mathbf{z_{s}}^{j},\mathbf{z_{t}}^{j}\\}$ and even the camera pose (see
Sec. 3.2). We show that CodeNeRF generalises well to render novel viewpoints,
demonstrating that the learnt priors help to complete unobserved geometry and
texture patterns (see Figs. 10 and 13).

Volume Rendering We follow [15] and use classical volumetric rendering
techniques to render the color of image pixels by aggregating the color and
the occupancy density at N uniform samples from the near to far bounds for the
camera ray $r(t)=c+t\mathbf{d}$ traced through each pixel. The estimated color
$C(r)$ of the pixel can be expressed as:

$\hat{C}(r)=\sum_{i=1}^{\text{N}}T_{i}(1-\exp(-\sigma_{i}\delta_{i}))\mathbf{c}_{i}\vspace{-.1cm}$
(3)

where $\delta_{i}=t_{i+1}-t_{i}$ is the distance between adjacent samples and
$T_{i}=\exp(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j})$. This expression is fully
differentiable, and reduces to the traditional alpha compositing with alpha
values. NeRF [15] adopts a 2 stage approach, a coarse and a fine network, with
the fine network using importance sampling based on the values of $T_{i}$ from
the coarse network to sample closer to the surface.

###  3.1 Training CodeNeRF

For each image in the training set
$\mathcal{V}=\\{\mathcal{I}_{i},\text{K}_{i},\text{T}_{i}\\}_{i=1}^{\text{N}}$,
at each training iteration we sample a batch of 4094 rays using the intrinsic
and extrinsic parameters. We then sample $64$ points along each ray and feed
them to the MLP $F_{\Theta}$ together with the current estimates of shape and
texture codes. $F_{\Theta}$ provides the occupancy density $\sigma$ and the
RGB color $\mathbf{c}$ and volume rendering is used to aggregate colors along
each ray. Training is supervised using the photometric loss along with a
regularization loss used as a prior over the latent vectors (see Eq.
$\eqref{eq:loss_train}$). We train the model with AdamW [13] optimizer with
initial learning rates 1e-4 for the network parameters and 1e-3 for the latent
vectors.

$\begin{split}\mathcal{L}&(\Theta,\\{\mathbf{z_{s}}^{i},\mathbf{z_{t}}^{i}\\}^{\text{M}})=\sum_{r\in\mathcal{R}}||\hat{C(r)}-C(r)||_{2}^{2}\\\
\min_{\Theta,\\{\mathbf{z_{s}}^{i},\mathbf{z_{t}}^{i}\\}^{\text{M}}}\mathcal{L}&(\Theta,\\{\mathbf{z_{s}}^{i},\mathbf{z_{t}}^{i}\\})+\frac{1}{\nu^{2}}(||\mathbf{z_{s}}^{i}||^{2}_{2}+||\mathbf{z_{t}}^{i}||^{2}_{2})\end{split}$
(4)

###  3.2 Inference Optimization

At inference time, given a single input image of an unseen object, the trained
model can be used to optimize the latent vectors by minimizing the photometric
loss between the rendered and observed pixels (Eq. 5), while keeping the
weights of the neural network fixed. Moreover, unlike previous methods [24,
37] we show that CodeNeRF does not require known camera pose and its
parameters can also be optimized jointly with the latent codes.

Camera Pose Optimization: CodeNeRF backprojects pixels into 3D rays using the
camera to world transformation matrix. Similarly to NeRF [15], we require
known camera matrices to train the model. However, once the model has been
learnt, at inference time, unlike others [24, 15, 37], we do not require known
camera viewpoint and we optimize the rotation and translation parameters
jointly with the latent embedding vectors, by obtaining the gradients over the
input points and ray directions, in a similar way to a differentiable
renderer.

Figure 5: Camera viewpoint parametrization. At test time, CodeNeRF optimizes
camera pose jointly with latent codes.

Similarly to other object-category 3D reconstruction methods, we assume a
simplified camera model, where the world coordinate system is centred on the
3D object, the camera faces its origin, and the up vector is aligned with the
$z$ axis. In this case, R and t can be parametrized via azimuth $\phi$,
elevation $\theta$ and distance to the camera $\rho$, as shown in Fig. 5. The
camera to world transformation matrix can be written as
$\textbf{T}_{cw}=\big{(}\begin{smallmatrix}\textbf{R}^{T}&\textbf{p}\\\
0&1\end{smallmatrix}\big{)}$ where:

$\textbf{R}=\begin{pmatrix}-\sin\phi&\cos\phi&0\\\
-\sin\theta\cos\phi&-\sin\theta\sin\phi&\cos\theta\\\
\cos\theta\cos\phi&\cos\theta\sin\phi&\sin\theta\\\ \end{pmatrix}$

and
$\textbf{p}=(\rho\cos\theta\cos\phi,\rho\cos\theta\sin\phi,\rho\sin\theta)$ is
the camera location. At inference time, given an reference image
$\mathcal{I}_{i}$, the camera pose and the sampled rays $\gamma_{i}$ can be
expressed in terms of the camera parameters azimuth $(\phi,\theta,\rho)$ and
optimized via using the gradients.

Test time optimization: We minimize the photometric loss $\eqref{eq:loss_inf}$
jointly with respect to shape and texture codes and camera parameters (fixing
the decoder parameters $\Theta$), using the AdamW optimizer.

$\min_{\mathbf{z_{s}}^{i},\mathbf{z_{t}}^{i},\rho_{i},\theta_{i},\phi_{i}}\mathcal{L}(\mathbf{z_{s}}^{i},\mathbf{z_{t}}^{i},\rho_{i},\theta_{i},\phi_{i})+\frac{1}{\nu^{2}}(||\mathbf{z_{s}}^{i}||^{2}_{2}+||\mathbf{z_{t}}^{i}||^{2}_{2})$
(5)

Latent vectors are initialized with the mean vector of the trained embeddings,
and an initial learning rate of 1e-2 is used. The initial learning rates for
the camera parameters are 1e-2, 1e-1 and 1e-1 respectively.

Enforcing disentanglement: To illustrate the link between good architecture
choices and disentanglement, we trained two alternative models (M1/M2). In M1,
$\gamma_{d}(d)$ is given as input to the first layer of $F^{s}_{\theta_{s}}$
jointly with 3D point position $\gamma_{x}(x)$ (similarly to PixelNeRF). In
M2, $z_{s}$ and $z_{t}$ are given as input to $F^{s}_{\theta_{s}}$, which
amounts to using a single embedding. We then perform texture editing keeping
the shape code fixed. Fig. 6 shows how only CodeNeRF fully disambiguates shape
and texture. M1 fails to synthesise the correct texture while M2 has unwanted
changes in shape.

Figure 6: Shape/Texture disentanglement: Alternative architectures fail to
disentangle shape and texture.

##  4 Experimental Evaluation

| 1-view | 2-view
---|---|---
| PSNR | SSIM | PSNR | SSIM
Chairs |  |  |  |
GRF [28] | 21.25 | 0.86 | 22.65 | 0.88
TCO [26] | 21.27 | 0.88 | 21.33 | 0.88
dGQN [5] | 21.59 | 0.87 | 22.36 | 0.89
ENR [4] | 22.83 | - | - | -
SRN [24] | 22.89 | 0.89 | 24.48 | 0.92
PixelNeRF [37] | 23.72 | 0.91 | 26.20 | 0.94
CodeNeRF (GT pose) | 23.66 | 0.90 | 25.63 | 0.91
CodeNeRF | 22.39 | 0.87 | - | -
CodeNeRF ($-$ outliers) | 23.11 | 0.89 | - | -
Cars |  |  |  |
SRN [24] | 22.25 | 0.89 | 24.84 | 0.92
ENR [4] | 22.26 | - | - | -
PixelNeRF [37] | 23.17 | 0.90 | 25.66 | 0.94
CodeNeRF (GT pose) | 23.80 | 0.91 | 25.71 | 0.93
CodeNeRF | 22.73 | 0.89 | - | -
CodeNeRF ($-$ outliers) | 23.17 | 0.90 | - | -
Table 2: Quantitative evaluation on ShapeNet-SRN Figure 7: Qualitative results
of novel view synthesis of unseen objects from a single input image with
CodeNeRF (GT pose), a variant of CodeNeRF which assumes known camera pose at
test time. Rightmost column shows input view.  Figure 8: Test-time joint
optimization of camera pose and latent codes: Evolution of estimated
renderings from initialization (left), through intermediate optimization
iterations $5/10/50$, to final result after $299$ iterations. Figure 9: Novel
Shape/Texture/Pose Synthesis. Highlighted results show the renderings with
shape and texture codes corresponding to the reference views (obtained via
optimization). Others show results of varying shapes and textures simply by
editing corresponding latent codes.  Figure 10: Qualitative results on real-
world datasets (Stanford-Car [9] and Pix3D [25]) of CodeNeRF with joint
optimization of camera pose+latent codes. Evolution of estimated renderings
throughout the optimization from initialization through iterations $5/10/50$
and final result after $299$ iterations. Rightmost column shows input image.

We train our model on the ShapeNet-SRN dataset of ShapeNet [1] renderings for
two object categories: cars (3514) and chairs (6591), created by Sitzmann _et
al_. [24]. We train a different model for each category using the pre-defined
train/test split and conduct a quantitative evaluation on novel view synthesis
and few shot reconstruction tasks.

###  4.1 Quantitative Evaluation on SRN Benchmark

The ShapeNet-SRN benchmark [24] provides 251 test images on an Archimedeal
spiral per object in the test set. We follow the evaluation protocols provided
by SRN [24].

Baselines We compare quantitatively on the ShapeNet benchmark [24] with SRN
[24], ENR [4], PixelNeRF [37], and other current state of the art methods, on
the tasks of one-view and two-view 2D-supervised reconstruction. Note that
competing approaches require known camera pose at inference time. We compare
with three variants of our approach: CodeNeRF (GT pose) assumes known camera
pose at test time, CodeNeRF is our full approach with joint optimization of
camera pose and shape/texture codes (299 iterations) and CodeNeRF ($-$
outliers), which is CodeNeRF but without counting results that resulted in an
outlier (rotation error above $5^{\circ}{}$ or translation error higer than
$3\%$).

Metrics We use PSNR(Peak Signal to Noise Ratio) and SSIM(Structural Similarity
Index Measure) [30] to evaluate the quality of the rendered images.

Synthesis Results Table 2, shows a full comparison of the results on 1-view
and 2-view reconstruction. CodeNeRF (GT pose) – which assumes known camera
pose, for a fairer comparison with its competitors that need it as input –
outperforms PixelNeRF for cars and comes very close on chairs. Even when
CodeNeRF estimates the unknown pose at test time, along with the latent codes,
the performance does not degrade much. When pose estimation outliers are
removed the performance is close to PixelNeRF. Qualitative results for 1-view
reconstruction with CodeNeRF (GT pose) are shown in Fig. 7 while Fig. 8 shows
results for CodeNeRF with camera pose+latent codes optimization. Clearly,
rendered images become sharper when the ground truth pose is used, but as Fig.
8 shows, even when pose and latent codes are initialized far from the ground
truth, CodeNeRF converges to good estimates.

Editing Novel Shapes and Textures After the test-time optimization has taken
place, CodeNeRF provides full control over the synthesis process as object
shapes and textures can be edited simply by varying the corresponding latent
codes. This is possible given that viewpoint, shape and texture are fully
disentangled in our representation. As shown in Fig. 9, we can easily fix one
while changing the other.

Evaluation of Camera Viewpoint Estimation: At test time, CodeNeRF optimizes
jointly the camera pose and shape/pose latent codes. We run a quantitative
evaluation of the performance of the camera pose estimation on the ShapeNet-
SRN dataset. Fig. 11 (top) shows histograms with the distribution of pose
estimation errors. It is interesting to note that the initializations we used
for the camera pose parameters (shown in green) had rotation errors above
$30^{\circ}{}$ in $100\%$ of cases and were on average $80^{\circ}{}$ away
from GT. Despite this, our optimization converged in $85\%$ of cases. The
table in Fig. 11 (below) shows the numerical errors between the estimated and
ground-truth camera poses in terms of the percentage of rotation estimates
with errors below ${5^{\circ}}$ and ${10^{\circ}}$, and the percentage of
translation estimates with relative errors below ${3\%}$ and ${5\%}$.

Figure 11: Pose estimation evaluation. Error distribution histograms: rotation
(top-left), translation (top-right). Initialization error in green. Numerical
evaluation (below).

Fig. 8 shows the initialization and illustrates the evolution of the
intermediate results obtained through the optimization (iterations 5/10/50),
and final result after 299 iterations. Even when the initial estimate is far
away from the ground truth, the optimization converges to good solutions.

###  4.2 Qualitative Comparisons

Fig. 12 shows examples comparing CodeNeRF with PixelNeRF [37] and SRN [24] on
new view synthesis of objects from the ShapeNet-SRN test set. Since PixelNeRF
uses a pre-trained CNN to extract features, it renders sharper images when the
target view is close to the input view. SRN works better on occluded regions
than PixelNeRF as it learns a latent embedding that acts as a priors. CodeNeRF
achieves shape and texture disentanglement maintaining geometric and
appearance consistency in occluded regions. Only CodeNeRF can synthesise the
correct combination of shape and texture for the purple chair.

Figure 12: Qualitative comparisons with SRN [24] and PixelNeRF [37] (results
courtesy of A. Yu and V. Sitzmann).

###  4.3 Results on Real World Datasets

To understand if our model, trained only on synthetic renderings from
ShapeNet-SRN, can generalise well to real-world images, we perform an
evaluation on two datasets: Stanford-Car dataset [9] and Pix3D [25]. We
perform test time optimization given a single input image to estimate jointly
camera pose+latent codes. We then render the representation from unseen
viewpoints.

Pre-processing Similarly to PixelNeRF [37], we use Detectron2 [33] on the
Stanford-Car dataset to infer masks, then apply Gaussian blur and downscale
the images to $128\times 128$. For the real chairs in Pix3D [25], we carve out
the object with the provided ground-truth mask before downscaling.

Optimization of Camera Pose and Latent Vectors We show results of one-shot
reconstruction given a single input image with unknown camera pose. Since the
camera intrinsics are unknown, we assume the same focal length as in ShapeNet-
SRN. Fig. 10 shows the initialization and the intermediate results through
different iterations of the optimization. The optimization provides convincing
final renderings even when initialised from far away.

Figure 13: Qualitative evaluation of novel view synthesis on images from real-
world datasets (Stanford-Car [9] and Pix3D [25]). Input image is shown on the
rightmost column. Camera pose is estimated jointly with latent codes.

Shape, Texture and Viewpoint Editing Following one-shot optimization, we show
renderings of the reconstructed objects from novel viewpoints (see Fig. 13).
Fig. 14 illustrates results after editing shape and texture latent vectors.
The distinctive style of each object is preserved after shape or texture
transfer. The shape and texture priors learned by CodeNeRF help to complete
object shape and texture information that was not present in the input image.

Figure 14: Shape and texture editing on real images from the Stanford-Cars and
Pix3D datasets.

Explicit mesh reconstruction CodeNeRF implicitly represents the 3D structure
of reconstructed objects. To obtain an explicit mesh representation, we feed
the centre of of 2563 voxels to the MLP, and obtain their volume density
values $\sigma$. Marching cubes can then be used to find the mesh vertices.
Casting rays along vertex normals we can associating colors to the vertices.
Fig. 15 shows the input image and the reconstructed meshes for a real image
from the Stanford-Car dataset and a synthetic car from ShapeNet.

Figure 15: Reconstructing 3D meshes: CodeNeRF allows 3D reconstruction of a
mesh from a single input image.

##  5 Conclusion

We have presented CodeNeRF, a neural radiance field that learns to disentangle
shape and appearance by jointly learning separate latent embeddings for shape
and texture, and an MLP that maps continuous input locations and ray
directions to density and colour information. At inference time, given a
single input image CodeNeRF can estimate its associated camera pose and latent
codes. We show extensive results of one-shot reconstruction on the SRN
benchmark and on real world images (Stanford-Cars and Pix3D) that demonstrate
that CodeNeRF learns powerful scene priors that enable full control over the
synthesis process.

Acknowledgements: Research presented here has been supported by funding from
Cisco to the UCL AI Centre.

## References

  * [1] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [2] Zhiqin Chen.  IM-NET: Learning implicit fields for generative shape modeling.  PhD thesis, Applied Sciences: School of Computing Science, 2019.
  * [3] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In European conference on computer vision, pages 628–644. Springer, 2016.
  * [4] Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan.  Equivariant neural rendering.  In International Conference on Machine Learning, pages 2761–2770. PMLR, 2020.
  * [5] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al.  Neural scene representation and rendering.  Science, 360(6394):1204–1210, 2018.
  * [6] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9784–9794, 2019.
  * [7] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  A papier-mâché approach to learning 3d surface generation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216–224, 2018.
  * [8] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3d mesh renderer.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [9] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [10] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen.  Differentiable monte carlo ray tracing through edge sampling.  ACM Transactions on Graphics (TOG), 37(6):1–11, 2018.
  * [11] Yiyi Liao, Simon Donne, and Andreas Geiger.  Deep marching cubes: Learning explicit surface representations.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2916–2925, 2018.
  * [12] Chen-Hsuan Lin, Oliver Wang, Bryan C Russell, Eli Shechtman, Vladimir G Kim, Matthew Fisher, and Simon Lucey.  Photometric mesh optimization for video-aligned 3d object reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 969–978, 2019.
  * [13] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [14] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [15] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  arXiv preprint arXiv:2003.08934, 2020.
  * [16] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504–3515, 2020.
  * [17] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  * [18] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [19] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv preprint arXiv:2007.08501, 2020.
  * [20] Edoardo Remelli, Artem Lukoianov, Stephan R. Richter, Benoit Guillard, Timur M. Bagautdinov, P. Baqué, and P. Fua.  Meshsdf: Differentiable iso-surface extraction.  ArXiv, abs/2006.03997, 2020.
  * [21] Martin Runz, Kejie Li, Meng Tang, Lingni Ma, Chen Kong, Tanner Schmidt, Ian Reid, Lourdes Agapito, Julian Straub, Steven Lovegrove, et al.  Frodo: From detections to 3d objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14720–14729, 2020.
  * [22] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  arXiv preprint arXiv:2007.02442, 2020.
  * [23] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019.
  * [24] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations, 2020.
  * [25] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3d: Dataset and methods for single-image 3d shape modeling.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [26] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Single-view to multi-view: Reconstructing unseen views with a convolutional network.  CoRR abs/1511.06702, 1(2):2, 2015.
  * [27] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3405–3414, 2019.
  * [28] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d scene representation and rendering.  arXiv preprint arXiv:2010.04595, 2020.
  * [29] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 52–67, 2018.
  * [30] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [31] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  Nerf $--$: Neural radiance fields without known camera parameters.  arXiv preprint arXiv:2102.07064, 2021.
  * [32] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  In Advances in Neural Information Processing Systems, pages 82–90, 2016.
  * [33] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [34] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  * [35] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.  In Advances in Neural Information Processing Systems, pages 492–502, 2019.
  * [36] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin.  inerf: Inverting neural radiance fields for pose estimation.  arXiv preprint arXiv:2012.05877, 2020.
  * [37] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  arXiv preprint arXiv:2012.02190, 2020.


SHOARI NEJAD et al

*Amin Shoari Nejad,

# Vector Time Series Modelling of Turbidity in Dublin Bay

Amin Shoari Nejad  Gerard D. McCarthy  Brian Kelleher  Anthony Grey  Andrew
Parnell  Hamilton Institute, Insight Centre for Data Analytics, Maynooth
University, Kildare, Ireland  ICARUS, Department of Geography, Maynooth
University, Maynooth, Ireland  Organic Geochemical Research Laboratory, Dublin
City University, DCU Glasnevin Campus, Dublin 9, Ireland
<EMAIL_ADDRESS>
###### Abstract

[Abstract]Turbidity is commonly monitored as an important water quality index.
Human activities, such as dredging and dumping operations, can disrupt
turbidity levels and should be monitored and analyzed for possible effects. In
this paper, we model the variations of turbidity in Dublin Bay over space and
time to investigate the effects of dumping and dredging while controlling for
the effect of wind speed as a common atmospheric effect. We develop a novel
Vector Auto-Regressive Conditional Heteroskedasticity (VARCH) approach to
modelling the dynamical behaviour of turbidity over different locations and at
different water depths. We use daily values of turbidity during the years
2017-2018 to fit the model. We show that the results of our fitted model are
in line with the observed data and that the uncertainties, measured through
Bayesian credible intervals, are well calibrated. Furthermore, we show that
the daily effects of dredging and dumping on turbidity are negligible in
comparison to that of wind speed.

###### keywords:

Bayesian, Vector Autoregression, Turbidity

##  1 Introduction

Studying the variables affecting turbidity is of importance in maintaining
coastal ecosystem health. Turbidity is an index for water clarity which
measures how suspended solids in water hinder the transmission of light 6.
There are many sources of suspended solids including: phytoplankton; particles
from coastal erosion; re-suspended bed sediments; organic detritus from
streams; and excessive algae growth 3. Variability in water turbidity
influences the transportation dynamics and distribution of nutrients,
contaminants, and biological production 25, 16, 8, 10, 11. Water turbidity is
an important habitat factor in many estuarine systems, and changes in it can
have a significant impact on management decisions such as the dredging of
ports and canals 2.

Our goal in this paper is to evaluate the variations of turbidity in Dublin
Bay explained by dredging and dumping operations when controlling for the
effect of wind speed, which is an important atmospheric contributor. Dublin
has a long history of difficult access for ships to the port area due to
sandbanks at the mouth of the port 9. To solve this problem regular dredging
operations have been carried over decades to remove unwanted waste as well as
dangerous accumulations of sediments from areas that ships use when entering
the port. The excavated materials from the dredging operations are dumped at a
more remote location in the bay.

There are relatively few studies focusing on water turbidity in Dublin bay. In
one example, 3 used frequentist statistical tests to show that turbidity can
be strongly influenced by vessel activity in Dublin bay using data collected
from a single location. By contrast, we take a broader approach and look at
multiple measuring sites simultaneously corresponding to both the sites where
sediment is dumped and dredged, whilst considering issues of turbidity down
the water column. We develop a novel Vector Auto Regressive Conditional
Heteroskedasticity (VARCH) model to control for the spatio-temporal structure
using turbidity data measured by five buoys installed at different locations
in the bay. Our model combines three well known approaches: Bayesian time
series modelling, which has been successfully applied to many problems for
example see 5, 14, 22; vector autoregression (VAR), originally introduced by
23 and widely used in macroeconomics, causal inference, and forecasting 15, 1,
21; and autoregressive conditional heteroscedasticity (ARCH) models that have
been employed in many forecasting applications see e.g. 7. In particular,
Bayesian time series modelling with a VAR structure has been used before to
model spatio-temporal data. For example, 21 used a Bayesian VAR approach to
model stream networks time series that were collected from different
locations. In this paper, we extend the Bayesian VAR models by relaxing the
assumption of constant variance over time, and letting the data generating
process incorporate time and space varying variance. To the best of our
knowledge, there have been no attempt in building this type of model for real
world data which are most likely heteroscedastic in nature.

We organise our paper as follows. In Section 2, we describe the data we use in
our study. In Section 3, we explain our modelling framework. In Section 4, we
discuss our findings including plots of the model outputs. We summarise the
paper in Section 5 by considering the strengths and weaknesses of our approach
and potential areas for future research.

##  2 Data description

Water turbidity levels are measured in Nephelometric Turbidity Units (NTU)
which calculate the amount of light reflected through a set of suspended
particles. Our dataset contains measurements of water turbidity in NTU at five
different locations, four of which take measurements at a single depth and are
located throughout the channel from the River Liffey towards Dublin Bay where
dredging takes place. The fifth buoy takes measurements at three different
levels of the water column and is located approximately 10 kilometres away
from Dublin port at the location where the sediments are dumped. Figure 1
shows the locations of the buoys in the bay.

Figure 1: Buoys measuring turbidity in Dublin Bay. We use the same numbering
scheme when referring to each site throughout the paper. Buoys 4 to 7 are
potential dredging sites, whilst the sediment is dumped at dumpsite.

Turbidity measurements are recorded every 15 minutes by the buoys, but for our
analysis we aggregated the raw data into daily averages. This allowed us to
focus on the impact of dredging whilst removed short term fluctuations (e.g.
that of tides) or the instantaneous impact of vessels arriving or leaving from
the port. The aggregation resulted in a total of 488 daily observations per
buoy from 31/08/2017 to 31/12/2018. However there are some periods with
missing data which seems to be due to equipment failure (e.g. discharged
batteries) and gives rise to data gaps when working with our sensor data. A
plot of the raw data with missing values at one buoy is provided in Figure 2.
Additionally, we use wind speed data measured at Dublin Airport, provided by
19 for the same period as turbidity data, to control for its effect on
turbidity.

Figure 2: Daily measurements of turbidity at Tolka from 31/08/2017 to
31/12/2018.

##  3 Modelling procedure

In this section we describe the general modelling framework that we follow to
build a dynamic spatio-temporal model that describes the response of turbidity
to a variety of environmental factors. We then provide specific variations on
this template to create four different models which we use for fitting on the
data. We denote $Y_{t}$ as an $S$-vector of turbidity measurements at time $t$
where $S$ is the number of locations (or equivalently buoys), $s=1,2,...,S$
represent the locations and times $t\ (t=1,2,...,T)$. We write the model
hierarchically in two main layers as:

$\displaystyle Y_{t}\mid M_{t},\Sigma_{t}$ $\displaystyle\sim$ $\displaystyle
MVN\left(M_{t},\Sigma_{t}\right)$ (1) $\displaystyle M_{t}$ $\displaystyle=$
$\displaystyle A+\sum_{j=1}^{P}X_{jt}\circ\beta_{j}+U_{t}$ (2)

where $M_{t}$ is the process mean and $\Sigma_{t}$ is the covariance matrix at
time $t$. $A$ is an intercept vector, $X_{jt}$ is an $S$-vector of covariate
values associated with covariate $j=1,\ldots,P$, $\beta_{j}$ is an $S$-vector
of fixed effects associated with covariate $j$, and $U_{t}$ is a spatio-
temporal structured effect. We use $\circ$ to denote the Hadamard product.

The four different structures we consider for fitting the model involve
specifying structures for the latent effects $U_{t}$ and the covariance matrix
$\Sigma_{t}$. We specify prior distributions associated with these models in
the section below following their definition.

Model 1

An ARCH structure with varying
$\Sigma_{t}=diag\\{\sigma_{t,1}^{2},...,\sigma_{t,s}^{2}\\}$ and:

$\displaystyle\sigma_{t,s}^{2}$ $\displaystyle=$
$\displaystyle\theta_{1,s}+\theta_{2,s}Y_{t-1,s}^{2}$ (3) $\displaystyle
U_{t}$ $\displaystyle=$ $\displaystyle\Phi U_{t-1}$ (4)

with $\Phi=\mbox{diag}\\{\phi_{1},...,\phi_{s}\\}$ being an $S\times S$
diagonal matrix of autocorrelation parameters and
$U_{t-1}=Y_{t-1}-A-\sum_{j=1}^{P}X_{j,t-1}\circ\beta_{j}$.

Model 2

A VAR model with a fixed time-invariant covariance matrix given an inverse-
Wishart $\mathcal{IW}$ prior:

$\displaystyle\Sigma$ $\displaystyle\sim$
$\displaystyle\mathcal{IW}^{-1}(\Psi,\nu)$ (5) $\displaystyle U_{t}$
$\displaystyle=$ $\displaystyle\Phi U_{t-1}$ (6)

with $\nu$ and $\Psi$ as fixed hyper-parameters (we use $\nu=14$ and $\Psi=I$
in our example), and where now $\Phi$ is a full rank matrix:

$\Phi=\begin{bmatrix}\phi_{1,1}&\ldots&\phi_{1,s}\\\ \vdots&\ddots&\vdots\\\
\phi_{s,1}&\ldots&\phi_{s,s}\\\ \end{bmatrix}.$

Model 3

A VARCH model which has both the full rank $\Phi$ from model 2 and the time-
varying error covariance matrix of model 1.

Model 4

A VARICH integrated model that uses the difference in the latent spatio-
temporal effects as well as the time-varying error covariance matrix:

$\displaystyle U_{t}-U_{t-1}=\Phi(U_{t-1}-U_{t-2})$ (7)

where as above the matrix $\Phi$ is of full rank.

To complete the model we need to specify prior distributions for all
parameters. We aim to use informative priors for those where we have some
degree of information, and use weakly informative and non-informative priors
for the remainder. In the below we outline our prior specification for the
most complex of the models we fit, model 4, though identical priors were used
in the simpler models which corresponds to setting some of the parameter
values to zero in a nested model structure.

Our covariates contained in $X_{j,t}$ consist of values associated with
dumping (binary yes/no at each site and time), dredging (binary yes/no at each
site and time), and wind speed (knots). The latter measure is common across
all sites but specified separately to allow for different effects at the
different sites. It is helpful, for prior specification, to consider the
regression parameters $\beta$ in terms of their individual scalar components
$[\beta_{dredge,s},\beta_{dumping,s},\beta_{wind,s}]$ at site $s$.

Since the dumpsite and the dredging site are approximately 10 kilometers away
from each other it seems reasonable to assume that dumping has no effect on
turbidity at sites which are used for dredging and vice versa. Setting these
values exactly to zero seemed to produce convergence problems in our Hamilton
Monte Carlo Algorithm. We found that a prior of $N(0,0.3^{2})$ was sufficient
to remove the effects whilst still producing reasonable posterior convergence
statistics. The full set of priors we used for these values is:

$\beta_{dredge,s}\sim\left\\{\begin{array}[]{l}N\left(0,25^{2}\right),\forall
s:s\in\mbox{Dredging site}\\\ N\left(0,0.3^{2}\right),\forall
s:s\in\mbox{Dumpsite}\end{array}\right.$

$\beta_{dump,s}\sim\left\\{\begin{array}[]{l}N\left(0,25^{2}\right),\forall
s:s\in\mbox{Dumpsite}\\\ N\left(0,0.3^{2}\right),\forall s:s\in\mbox{Dredging
site}\end{array}\right.$

The value of 25 for the prior standard deviation of these parameters was
chosen to match the approximate range of the turbidity data and was
unrestricted in sign to allow for both positive and negative effects of
dumping or dredging on turbidity.

For the $\Phi$ matrix we assume that off-diagonal values are most likely
smaller than the diagonal ones since we believe the main autoregressive effect
to be within site, and that cross site effects need to be strongly supported
by the data to persist into the posterior distribution. For the diagonal terms
we focus most of the prior MASS in the range (-1,1) so that the model selects
for stationary behaviour, though non-stationarity can be found if the data are
indicative of such phenomena. We thus use:

$\Phi_{ij}\sim\left\\{\begin{array}[]{l}N(0,0.1^{2}),\mbox{ if }i\neq j\\\
N(0,0.5^{2}),\mbox{ otherwise}\end{array}\right.$

For the remaining parameters we set:

$\displaystyle A_{s}$ $\displaystyle\sim$ $\displaystyle N(0,100^{2})$
$\displaystyle\beta_{wind,s}$ $\displaystyle\sim$ $\displaystyle N(0,1)$
$\displaystyle\theta_{1,s}$ $\displaystyle\sim$ $\displaystyle TN_{0}(0,1)$
$\displaystyle\theta_{2,s}$ $\displaystyle\sim$ $\displaystyle Beta(1,5)$

where $TN_{a}$ refers to the truncated normal distribution with minimum value
$a$. All these are expected to be weakly informative, guiding the model
towards sensible values whilst letting the data provide the majority of the
information . Turbidity in our dataset ranges between 0-130 (NTU), so the
prior values chosen for $A$, $\beta_{dredge}$, $\beta_{dump}$, and
$y_{missing}$ are considered to be uninformative with respect to this range.
The same thing is true for $\beta_{wind}$ knowing that wind speed can reach as
high as 70 knots during storms, and so a high value of 2 in units of NTU per
knot (the units of $\beta_{wind}$ seems reasonable).

As a final remark on priors we note that many of the turbidity values across
sites are missing. We assume that these values are missing at random MAR; 17
and impute them as part of the model fitting step by treating them as
parameters to be estimated. When using the likelihood given above we found
that we struggled to produce a posterior with finite variance so we added the
extra prior constraint $\ y_{missing}\sim TN_{0}^{100}(0,50^{2})$, a truncated
normal between 0 and 100, which seemed to stabilise the missing value
estimates.

In summary, model 1 provides a baseline univariate autoregressive model with
time changing variance. A more basic constant variance model was also
attempted but not shown here due to poor performance. Model 2 tests whether a
richer full rank vector mean structure improves the fit at the expense of the
changing variance. Model 3 combines both the full vector autoregression with
the time changing variance. Finally, model 4 introduces a difference in the
latent parameters to capture any potential non-stationarity in the mean. Below
we fit each of these models to the data described in Section 2, and use a
combination of posterior predictive distributions, information criteria, and
plots of the posterior distributions of the parameters to determine the
optimal models which we use for interpreting our findings.

##  4 Results

In this section, we report the results of fitting the models described in
Section 3 to the turbidity data. We summarise the estimated effects of
dredging and dumping operations (recall these are binary variables) and
account for the wind speed effect by including the daily wind speed measured
in knots. We compare the different models according to their fit to the data,
and interpret the best fitting model with a view to obtaining a better
understanding of turbidity behaviour in Dublin bay.

###  4.1 Model fitting and comparison

We fit the models using R 20 and the Stan modelling framework 24. This
approach uses Hamiltonian Monte Carlo to update all parameters simultaneously
and aims to rapidly converge to the posterior distribution. Through repeated
fitting of the models we found that using 1000 iterations, with a warm-up
period of 200 iterations, produced acceptable results. We checked convergence
using the R-hat diagnostic 4, 12 which were all around the target value of 1
at convergence.

To compare between the models, we use a combination of visual checks where we
plot the one-step ahead forecasts from the model (defined via $M_{t}$) against
the true data values, the posterior predictive distribution from the model,
and more formal methods. In particular we use the Widely Applicable
Information Criterion (WAIC, 27) and the Leave-One-Out Information Criterion
(LOO-IC, 26) which penalise the likelihood of the model fit based on the
complexity of the model. These two information criteria have the added
advantage of being easily implemented in R and providing an uncertainty
estimate on the value itself.

Figure 3 shows the estimated WAIC and LOOIC values for the four models. The
VARICH and VARCH models have the lowest WAIC and LOOIC values indicating
better fits. However whilst the mean values of WAIC for the VARICH model are
slightly lower there is no clear difference between them. The VARICH model has
no extra complexity compared to VARCH, i.e. there are no extra parameters to
estimate. Furthermore we computed the spectral radius of the posterior mean of
$\Phi$ for both models; VARICH gave 0.33 compared to 0.98 for VARCH, which
indicates that the VARICH model seems to have removed some of the non-
stationarity present in the VARCH formulation. We thus use the VARICH model to
create our further results.

Figure 3: WAIC and LOOIC values for the four fitted models with their
associated standard errors.

Figure 4 shows the one step ahead fitted forecasts of turbidity from the
VARICH model against the true data values. The expected values of the fit and
the observed values are shown with solid lines coloured in red and blue
respectively and the 95% credible intervals are shown with grey bands. As
mentioned in Section 2, the dataset has missing periods which are imputed for
each location by the model during the fitting process. The vector
autoregressive part of the VARICH model allows for drawing information for
each site using the available information from the other sites which
specifically helps regulate the uncertainty for the missing periods. As
expected, the uncertainty during high volatility periods grows as expected
through the dynamic structure applied to the variance.

Figure 4: One step ahead forecasts from the VARICH model vs observed values of
turbidity over time for the 7 buoys as labelled. Note the differing vertical
axis heights. The shaded periods indicate 95% credible intervals.

Figure 5 shows the posterior predictive distributions from the VARICH model
against the true values with vertical lines indicating the 95% uncertainty
intervals. On average the posterior prediction intervals cover 94.6% of the
data. The figure shows that the model can successfully retrieve the true
values of the turbidity in the dataset with well-calibrated uncertainty
estimation at the dumpsite and the dredging sites respectively.

Figure 5: Fitted values from the VARICH model versus observed values of
turbidity at different sites. The vertical bars indicate the 95% uncertainty
intervals which provide evidence of the coverage properties of the model.

###  4.2 Effects of covariates on turbidity

To determine the degree to which dumping and dredging operations affect
turbidity, we evaluate the posterior distribution of the fixed effects
$\beta$. Figure 6 shows the expected value of the dumping and dredging effects
respectively with their 95% credible intervals for different locations. All
the effects are seen to be close to zero. According to the figure, the
dredging at the Eastlink site has the greatest absolute value. However, the
effect is still small, and can be explained by the reduced vessel activities
during dredging operations as it is the most inland of the buoys we analyse
(cf Figure 1). An alternative explanation for this site is that freshwater
flow from the Liffey and Dodder rivers can vary with the catchment zone
precipitation; this impacts the salinity gradient in the water column and so
potentially exerts influence over turbidity 18.

Figure 6: Dumping effect (NTU/day) at different locataions with the 95%
credible interval (A) and dredging effect (NTU/day) at different locations
with the 95% credible interval (B).

Figure 7 shows the wind speed effects for the 7 buoys. These are measured in
NTU per knot and these wind effects can be more clearly identified than the
effects of dredging and dumping. The values are reasonably consistent but with
greater uncertainty at the lower positions in the dumping buoy, and a far
smaller effect at Eastlink, again likely due to its position in the bay. By
contrast, the Tolka buoy seems most influenced by wind and is the site that is
most far out to sea. The Tolka buoy is situated within the confines of the
estuary walls, adjacent to North Bull wall. This area of the estuary is
relatively shallow and at low tide is exposed to the wind.

Figure 7: Effect of average wind (NTU) at different locations and depths with
the 95% credible interval.

###  4.3 Influence of the autoregressive component

As a final part of the analysis, we examine the autoregressive coefficients
from the VARICH model. Figure 8 shows the posterior coefficients of $\Phi$
where we have separated out the diagonal values which indicate the influence
of the time series on itself from the off-diagonal elements which show the
influence of one site on another. The numbering of the sites is as shown in
Figure 1.

Of the diagonal elements, the dumpsite (middle) seems to have the most
dependence after accounting for the integration component. The other sites
have values close to zero after accounting for uncertainty. Of the off-
diagonal elements, some of these are well away from zero and provide for
interesting, if not entirely straightforward, interpretation. $\Phi_{34}$ is
the largest, corresponding to the relationship between dumpsite (bottom) and
buoy 4 (Tolka), which should perhaps be read in conjunction with their joint
time series behaviour as shown in Figure 4. Many of the other off-diagonal
elements show similar clear non-zero effect sizes though they are considerably
smaller than $\Phi_{34}$. These values provide evidence of cross site learning
in the time series model.

Figure 8: Coefficients of the $\Phi$ matrix with their 95% credible interval.
Diagonal values are shown in the top panel (a) and off-diagonal values are
shown in (b). The two subscripts indicate the parent and child relationship
respectively, so that $\Phi_{12}$ for example is the degree to which buoy 2
influences the time series of buoy 1. The numbers of the buoys follow the
labelling defined in Figure 1.

##  5 Discussion and conclusions

We have introduced a new set of models for understanding the behaviour of
turbidity in Dublin bay. The VARCH and VARICH models proposed in Section 3
allow for measuring the effects of multiple time series on each other, whilst
taking account of the known volatility changes in the time series. The
combination of Bayesian modelling, VAR and ARCH structures makes our VARICH
model a useful tool for flexible modelling of a wide range of real world
random processes in which spatial and temporal aspects are playing major
roles. Furthermore, the Bayesian approach allows for uncertainty
quantification of both the fixed effects and the posterior predictions of the
time series, whilst simultaneously imputing the missing values within the
series.

Our main finding has been that the dumping and dredging operations have
minimal effect on the turbidity levels, which seem to be more affected by wind
speed and previous values of the series. We thus suggest that, at an aggregate
daily level, there is minimal effect of dredging on the turbidity levels in
Dublin bay. The models we produced seem to fit the data well and the results
make physical sense according to the location of the buoys in the bay. A
longer time series and a more complete record would add further weight to our
conclusions.

The main challenge in fitting our model has been the computational cost of the
Hamiltonian MCMC. As the number of time series increases, the well known “big
n" problem of the VAR models kicks in. We have used informative priors to cope
with this problem for our grouped time series, for which we have reasonable
knowledge about the site connections to induce some level of sparsity in the
$\Phi$ matrix structure through priors with smaller variance for the off-
diagonal/cross-site effects. This is similar to 21, who introduced zeros into
the $\Phi$ matrix of their model for the parameters where any dependence was
implausible. In the future, we may also follow 28 who suggested a
regularization prior to make the $\Phi$ matrix sparse for faster computation.
This approach might be useful for superior estimation of the autoregression
coefficients for large VAR models. A related alternative would be the recent
work of 13 who provide stability inducing priors to ensure that the VAR
coefficients remain in a stationary region of parameter space.

##  6 Acknowledgements

We would like to thank Dr Sarah Heaps for her very help comments and
assistance with the coding of some of the models. This work was supported by
an SFI Investigator award (16/IA/4520). In addition, Andrew Parnell’s work was
supported by: a Science Foundation Ireland Career Development Award
(17/CDA/4695); a Marine Research Programme funded by the Irish Government, co-
financed by the European Regional Development Fund (Grant-Aid Agreement No.
PBA/CC/18/01); European Union’s Horizon 2020 research and innovation programme
InnoVar under grant agreement No 818144; SFI Centre for Research Training in
Foundations of Data Science 18/CRT/6049, and SFI Research Centre award
12/RC/2289_P2. For the purpose of Open Access, the authors have applied a CC
BY public copyright licence to any Author Accepted Manuscript version arising
from this submission.

## References

  * Alola . 2021 alola2021Alola, AA., Uzuner, G. Akadiri, SS.  2021\.  Modeling tourism and fear nexus in G4 countries Modeling tourism and fear nexus in g4 countries.  Current Issues in Tourism24101333–1339.
  * Bever . 2018 bever2018Bever, AJ., MacWilliams, ML. Fullerton, DK.  2018\.  Influence of an observed decadal decline in wind speed on turbidity in the San Francisco Estuary Influence of an observed decadal decline in wind speed on turbidity in the san francisco estuary.  Estuaries and Coasts4171943–1967.
  * Briciu-Burghina . 2014 Briciu2014Briciu-Burghina, C., Sullivan, T., Chapman, J. Regan, F.  2014\.  Continuous high-frequency monitoring of estuarine water quality as a decision support tool: a Dublin Port case study Continuous high-frequency monitoring of estuarine water quality as a decision support tool: a dublin port case study.  Environmental monitoring and assessment18995561-5580.
  * Brooks Gelman 1998 rhat1Brooks, S. Gelman, A.  1998\.  General methods for monitoring convergence of iterative simulations General methods for monitoring convergence of iterative simulations.  Journal of computational and graphical statistics74434-455.
  * Chao . 2020 chao2020Chao, F., Gerland, P., Cook, AR. Alkema, L.  2020\.  Global estimation and scenario-based projections of sex ratio at birth and missing female births using a Bayesian hierarchical time series mixture model Global estimation and scenario-based projections of sex ratio at birth and missing female births using a bayesian hierarchical time series mixture model.  arXiv preprint arXiv:2006.07101.
  * Davies-Colley Smith 2001 davies2001Davies-Colley, R. Smith, D.  2001\.  Turbidity, suspended sediment, and water clarity: a review Turbidity, suspended sediment, and water clarity: a review.  Journal of the American Water Resources Association3751085-1101.  https://doi.org/10.1111/j.1752-1688.2001.tb03624.x
  * Degiannakis Xekalaki 2004 degiannakis2004Degiannakis, S. Xekalaki, E.  2004\.  Autoregressive conditional heteroscedasticity (ARCH) models: A review Autoregressive conditional heteroscedasticity (arch) models: A review.  Quality Technology & Quantitative Management12271–324.
  * Díez-Minguito de Swart 2020 diez2020Díez-Minguito, M. de Swart, HE.  2020\.  Relationships Between Chlorophyll-a and Suspended Sediment Concentration in a High-Nutrient Load Estuary: An Observational and Idealized Modeling Approach Relationships between chlorophyll-a and suspended sediment concentration in a high-nutrient load estuary: An observational and idealized modeling approach.  Journal of Geophysical Research: Oceans1253no–no.
  * Dublin Port Company 2016 DPC2016Dublin Port Company.  2016\.  Report: Maintenance Dredging Campaign Report: Maintenance dredging campaign .
  * Ganju . 2020 ganju2020Ganju, NK., Testa, JM., Suttles, SE. Aretxabaleta, AL.  2020\.  Spatiotemporal variability of light attenuation and net ecosystem metabolism in a back-barrier estuary Spatiotemporal variability of light attenuation and net ecosystem metabolism in a back-barrier estuary.  Ocean Science163593–614.
  * Ge . 2020 GE2020Ge, J., Torres, R., Chen, C., Liu, J., Xu, Y., Bellerby, R.Ding, P.  2020\.  Influence of suspended sediment front on nutrients and phytoplankton dynamics off the Changjiang Estuary: A FVCOM-ERSEM coupled model experiment Influence of suspended sediment front on nutrients and phytoplankton dynamics off the changjiang estuary: A fvcom-ersem coupled model experiment.  Journal of Marine Systems204103292.  https://www.sciencedirect.com/science/article/pii/S0924796319304294 https://doi.org/10.1016/j.jmarsys.2019.103292
  * Gelman Rubin 1992 rhat2Gelman, A. Rubin, D.  1992\.  Inference from iterative simulation using multiple sequences Inference from iterative simulation using multiple sequences.  Statistical science74457-472.
  * Heaps 2022 heaps2022enforcingHeaps, SE.  2022\.  Enforcing stationarity through the prior in vector autoregressions Enforcing stationarity through the prior in vector autoregressions.  Journal of Computational and Graphical Statisticsjust-accepted1–24.
  * Hensman . 2013 hensman2013Hensman, J., Lawrence, ND. Rattray, M.  2013\.  Hierarchical Bayesian modelling of gene expression time series across irregularly sampled replicates and clusters Hierarchical bayesian modelling of gene expression time series across irregularly sampled replicates and clusters.  BMC bioinformatics1411–12.
  * Karlsson 2013 karlsson2013Karlsson, S.  2013\.  Forecasting with Bayesian vector autoregression Forecasting with bayesian vector autoregression.  Handbook of economic forecasting2791–897.
  * Kennish 1991 kennish1991Kennish, MJ.  1991\.  Ecology of estuaries: anthropogenic effects Ecology of estuaries: anthropogenic effects ( 1).  CRC press.
  * Little Rubin 2019 little2019statisticalLittle, RJ. Rubin, DB.  2019\.  Statistical analysis with missing data Statistical analysis with missing data ( 793).  John Wiley & Sons.
  * Lu . 2020 lu2020constraintsLu, T., Wu, H., Zhang, F., Li, J., Zhou, L., Jia, J.Wang, YP.  2020\.  Constraints of salinity-and sediment-induced stratification on the turbidity maximum in a tidal estuary Constraints of salinity-and sediment-induced stratification on the turbidity maximum in a tidal estuary.  Geo-Marine Letters405765–779.
  * Met Éireann 2022 winddataMet Éireann.  2022\.  Dublin Wind Data. Dublin wind data.  https://cli.fusio.net/cli/climate_data/webdata/dly532.csv
  * R Core Team 2022 RsoftR Core Team.  2022\.  R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing) R: A language and environment for statistical computing (r foundation for statistical computing) .
  * Santos-Fernandez . 2022 santos2022bayesianSantos-Fernandez, E., Ver Hoef, JM., Peterson, EE., McGree, J., Isaak, DJ. Mengersen, K.  2022\.  Bayesian spatio-temporal models for stream networks Bayesian spatio-temporal models for stream networks.  Computational Statistics & Data Analysis170107446.
  * Senf . 2017 senf2017Senf, C., Pflugmacher, D., Heurich, M. Krueger, T.  2017\.  A Bayesian hierarchical model for estimating spatial and temporal variation in vegetation phenology from Landsat time series A bayesian hierarchical model for estimating spatial and temporal variation in vegetation phenology from landsat time series.  Remote Sensing of Environment194155–160.
  * Sims 1980 sims1980Sims, CA.  1980\.  Macroeconomics and reality Macroeconomics and reality.  Econometrica: journal of the Econometric Society1–48.
  * Stan Development Team 2022 StanStan Development Team.  2022\.  CmdStanR: an R interface to Stan. R package version: 0.4.0.9001. Cmdstanr: an r interface to stan. r package version: 0.4.0.9001.
  * Tian . 2009 TIAN2009Tian, T., Merico, A., Su, J., Staneva, J., Wiltshire, K. Wirtz, K.  2009\.  Importance of resuspended sediment dynamics for the phytoplankton spring bloom in a coastal marine ecosystem Importance of resuspended sediment dynamics for the phytoplankton spring bloom in a coastal marine ecosystem.  Journal of Sea Research624214-228.  https://www.sciencedirect.com/science/article/pii/S1385110109000458 https://doi.org/10.1016/j.seares.2009.04.001
  * Vehtari . 2017 vehtari2017practicalVehtari, A., Gelman, A. Gabry, J.  2017\.  Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC Practical bayesian model evaluation using leave-one-out cross-validation and waic.  Statistics and computing2751413–1432.
  * Watanabe 2012 watanabe2012widelyWatanabe, S.  2012\.  A widely applicable Bayesian information criterion A widely applicable bayesian information criterion.  arXiv preprint arXiv:1208.6338.
  * Yan . 2021 yan2021Yan, Y., Huang, HC. Genton, MG.  2021\.  Vector Autoregressive Models with Spatially Structured Coefficients for Time Series on a Spatial Grid Vector autoregressive models with spatially structured coefficients for time series on a spatial grid.  Journal of Agricultural, Biological and Environmental Statistics1–22.