# PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-
Image Novel View Synthesis

Xianggang Yu1, Jiapeng Tang2, Yipeng Qin3, Chenghong Li1
Linchao Bao4, Xiaoguang Han1, Shuguang Cui1
1The Chinese University of Hong Kong, Shenzhen 2Technical University of Munich
3Cardiff University 4Tencent AI Lab

###### Abstract

We present PVSeRF, a learning framework that reconstructs neural radiance
fields from single-view RGB images, for novel view synthesis. Previous
solutions, such as pixelNeRF [66], rely only on pixel-aligned features and
suffer from feature ambiguity issues. As a result, they struggle with the
disentanglement of geometry and appearance, leading to implausible geometries
and blurry results. To address this challenge, we propose to incorporate
explicit geometry reasoning and combine it with pixel-aligned features for
radiance field prediction. Specifically, in addition to pixel-aligned
features, we further constrain the radiance field learning to be conditioned
on i) voxel-aligned features learned from a coarse volumetric grid and ii)
fine surface-aligned features extracted from a regressed point cloud. We show
that the introduction of such geometry-aware features helps to achieve a
better disentanglement between appearance and geometry,  _i.e_. recovering
more accurate geometries and synthesizing higher quality images of novel
views. Extensive experiments against state-of-the-art methods on ShapeNet
benchmarks demonstrate the superiority of our approach for single-image novel
view synthesis.

##  1 Introduction

Figure 1: Novel view synthesis from a single image. (a) Input image. (b) Novel
view synthesis results: pixelNeRF [66] (top) and Ours (bottom). (c) Surface
meshes extracted from predicted radiance fields: pixelNeRF [66] (top) and Ours
(bottom). By augmenting the 2D pixel-aligned features with complementary 3D
geometric features for radiance field prediction, we can synthesize higher
quality of novel views. A by-product of our approach is a cleaner implicit
surface mesh, due to the introduction of explicit geometric features.

Novel view synthesis is a long-standing problem in computer vision and
graphics, which plays a crucial role in various practical applications,
including gaming, movie production, and virtual/augment reality. Recently, it
has made great strides thanks to the advances in differentiable neural
rendering [35, 64], especially the neural radiance fields (NeRF) [29] that
simplifies novel view synthesis to an optimization problem over a dense set of
ground truth views.

Although achieving impressive results, the vanilla NeRF suffers from several
limitations: i) the dense views it strictly requires are not always available;
ii) it is slow in inference due to the long optimization process; iii) each
NeRF is dedicated to a specific scene and cannot be generalized to new ones.

Figure 2: Illustration of the feature ambiguity issue. The feature ambiguity
issues that originates from the many-to-one mapping between queries and their
corresponding pixel-aligned features. Two rays shot from points $t^{0}$ and
$t^{1}$ on target camera intersect the same ray shot from source camera. After
the pixel-aligned process, the two different intersecting points will be
projected to the same image coordinate $s^{0}$ on image-plane, obtaining the
same pixelwise feature.

To address these issues, follow-up works such as pixelNeRF [66], IBRNet [57],
and GRF [54], proposed to predict neural radiance fields in a feed-forward
manner. Taking pixelNeRF as an example, it tackles the shortcomings of NeRF by
extending its network to be conditioned on scene priors learnt by a
convolutional image encoder. These scene priors are represented by spatial
feature maps that allow the mapping from a pair of query spatial point and
viewing direction to their corresponding pixel-aligned features. In pixelNeRF,
such a mapping is implemented by standard camera projection and bilinear
interpolation. During inference, the scene priors are obtained via a forward
pass through the image encoder and thus allow fast novel view synthesis from a
single input view of diverse scenes. Although effective, pixelNeRF suffers
from feature ambiguity issues that originates from the many-to-one mapping
between queries and their corresponding pixel-aligned features. In other
words, pixelNeRF naively assigns the same pixel-aligned features to different
points in some novel view as long as these points overlap with each other in
the input view, which can cause confusion (Fig. 2).

To clarify such ambiguity issues, we propose to incorporate explicit geometry
reasoning and combine it with pixel-aligned features for radiance field
prediction. Specifically, we leverage the recent success in single-view 3D
reconstruction [7, 10, 13, 47, 36, 28, 4, 48] and inject rich geometry
information into radiance field prediction by incorporating geometry-aware
features of two shape representations: i) voxel-aligned features learned from
a coarse volumetric grid and ii) fine surface-aligned features extracted from
a regressed point cloud. Intuitively, such geometry-aware features augment
pixel-aligned features with additional “dimensions”, thereby allowing
previously ambiguous points to be separable. Furthermore, by constraining the
radiance field learning on these geometry-aware features, our method not just
synthesize higher quality images of novel views, but also recover more
accurate underlying geometries in radiance field, as witnessed in Fig. 1.

Our main contributions include:

  * •

We propose a novel approach of learning neural radiance fields from single-
view images jointly conditioned on pixel-, voxel-, surface-aligned features.

  * •

We design an efficient way to alleviate the feature ambiguity issue of solely
pixel-aligned features by incorporating explicit geometry reasoning via
single-view 3D reconstruction.

  * •

We propose a hybrid use of geometric features, including complementary coarse
volumetric features and fine surface features.

##  2 Related Work

Figure 3: Overview of our PVSeRF framework. Given a single input image, we
first 1) extract the spatial feature map using a fully convolutional image
encoder $E$, 2) learn a volumetric grid through volume generator
$G_{\mathbf{V}}$, and 3) regress a surface point set of the object through a
point set generator $G_{\mathcal{S}}$. From volumetric grid and surface point
set, we can learn voxel features and point-wise features. Then, for a 3D
location $\mathbf{X}$ and a target view direction $\mathbf{d}$, we query
pixel-, voxel-, and surface-aligned $\mathbf{f_{I}}$, $\mathbf{f_{V}}$,
$\mathbf{f}_{\mathcal{S}}$ from spatial feature map, voxel features and point-
wise features respectively. Next, the 3D location, view direction and all
corresponding features are directed into a MLP to predict density $\sigma$ and
radiance $\mathbf{r}$. Lastly, the volume rendering is used to accumulate the
radiance prediction of points on the same ray to compute the final color
values.

###  2.1 Novel view synthesis and Neural radiance field

The task of novel view synthesis aims to generate new views of a scene from
single or a set of sparse views. There are various kinds of approaches
dedicated to this problem. Traditional methods [12, 8, 24] choose to estimate
light fields and then render novel views. Recent years, with the advance of
deep neural networks (DNN), a plethora of models are designed to learn novel
view synthesis in an end-to-end manner. Pioneering methods [37, 51, 68]
consider it as a image-to-image transformation problem and directly utilize 2D
CNN to output novel views. These methods always cannot generate satisfactory
results for viewpoints that are largely deviated from the given view.

Later work explore 3D-aware image synthesis and solve the inverse rendering
problem via neural networks [45, 59, 34, 11, 22, 33, 69]. The common
characteristic of this line of literature is that they recover the explicit or
implicit 3D geometry and appearance properties first, then render novel views
at desired camera viewpoints by means of differentiable rendering techniques
[33] or generative models. Among these work, various 3D representations are
employed. DeepVoxels [45] represents 3D scene properties by low-resolution
volumetric feature grid lifted from 2D feature maps. Wiles  _et al_. [59] use
3D surface features that are learned from the point cloud unprojected from the
estimated depth map of the input view. Other approaches [34, 11, 22, 69] learn
implicit 3D embedding that can be used to generate novel views of the same
scene using unsupervised learning techniques.

Recently, witnessing the great success of neural radiance field (NeRF) [29],
there has been an explosion of NeRF-based approaches for novel view
synthesis[27, 26, 25, 31, 43, 65, 53, 39, 6, 66, 57, 54, 3]. There are two
divisions in the prevalence of NeRF: 1) the first track tries to train scene-
specific model for generating novel views of the scene [27, 26, 25, 31, 43,
65, 53, 39]. Specifically, they capture many diverse viewpoints of a scene,
and optimizing a neural radiance field for that scene. Despite synthesizing
high-fidelity novel views, these methods require longstanding optimization
process and cannot generalize to new scenes. 2) the second track attempts to
learn generalize neural radiance field across multiple scenes [6, 66, 57, 54,
3]. Among this, pixelNeRF [66] is the most relevant method to ours, which
learns the scene priors conditioned on the pixel-aligned features, and can
switch to new scenes flexibly. Although other methods [6, 57, 54, 3] can also
be applied to novel scene through a single forward pass, their methods are
equipped to multiple input views, while we focus on the more challenging
single-view input setting.

###  2.2 Single-view 3D Object Reconstruction

Given a single image containing a object, 3D object reconstruction aims to
recover the 3D geometry of the object. Traditional 3D reconstruction methods
[15, 18, 1, 52] need to find dense correspondence across multi-view at the
first, followed by the depth fusion stage. Recently, due to the establishment
of large-scale 3D model datasets, such as ShapeNet [2] and ModelNet [62], it
is popular to reconstruct complete 3D shape from a single image by utilizing
shape priors modeled by deep neural networks. It also achieved various degrees
of success by designing 3D shape decoders tailored for different shape
representations including voxel [7, 14], point cloud [10, 32], mesh [13, 36,
48], and implicit field [4, 28, 38, 48, 41, 49]. The voxel decoders [7, 61]
take advantages of conventional 3D convolution operations to generate
volumetric grids. The point decoders [10, 63] directly regress the coordinates
of 3D points. The mesh decoders mainly approximate a target shape by
performing template mesh deformation [13, 19, 36, 47, 48]. The neural implicit
functions [4, 28, 38, 48, 41] represent 3D surfaces by continuous functions
defined in 3D space. In this paper, we want to incorporate explicit geometry
reasoning into the process of single-view novel view synthesis by marrying
single-view 3D shape generators with a generic radiance field learning model.

##  3 PVSeRF

###  3.1 Overview

Given a single-view image $\mathbf{I}$ with its corresponding camera pose
$\mathbf{R}$ and intrinsic shape parameter $\mathbf{K}$, our PVSeRF aims to
learn a neural network for radiance field reconstruction:

$\sigma,\mathbf{r}=\mathrm{PVSeRF}(\mathbf{X},\mathbf{d};\mathbf{I},\mathbf{R},\mathbf{K})$
(1)

where $\mathbf{X}\in\mathbb{R}^{3}$ represents a 3D location,
$\mathbf{d}\in\mathbb{R}^{2}$ is a view direction, $\sigma$ is the volume
density at $\mathbf{X}$, and $\mathbf{r}$ is the predicted radiance (RGB
color) at $\mathbf{X}$ depending on the viewing direction $\mathbf{d}$. By
accumulating the $\sigma$ and $\mathbf{r}$ of multiple points sampled on the
ray defined by $\mathbf{X}$ and $\mathbf{d}$, we can obtain the color values
of all pixels in a target view image $\mathbf{I_{t}}$ via differentiable
rendering, thereby enabling novel view synthesis.

The distinct advantage of our PVSeRF is that it addresses the feature
ambiguity issue of pixelNeRF [66] by a novel geometric regularization using
both voxel- and surface-aligned features. As aforementioned, pixelNeRF’s
feature ambiguity issue stems from the fact that its network is solely
conditioned on the 2D pixel-aligned features where multiple query 3D points
are mapped to a single location. To clarify this ambiguity, we propose to
augment the 2D pixel-aligned features with complementary 3D geometric features
for radiance field construction. As Fig. 3 shows, in addition to the pixel-
aligned features, our method incorporates i) voxel-aligned and ii) surface-
aligned features into radiance field prediction. Specifically,

  * •

We follow [66] and extract the pixel-aligned features $\mathbf{f_{I}}$ of a
query point $\mathbf{X}$ by projecting it with $\mathbf{R}$ and $\mathbf{K}$
to the 2D image coordinates $x$, indexing the multi-scale feature maps of an
input image $\mathbf{I}$ extracted by a fully-convolutional image encoder $E$.

  * •

We extract the voxel-aligned features $\mathbf{f_{V}}$ of a query point
$\mathbf{X}$ by trilinearly interpolating $\mathbf{X}$ in a low-resolution
volumetric feature $\mathbf{F_{V}}$ learnt from the input image $\mathbf{I}$
using a volume generator $G_{\mathbf{V}}$. Note that $\mathbf{f_{V}}$ only
captures coarse geometry contexts of the scene due to the low-resolution
nature of $\mathbf{F_{V}}$.

  * •

To capture the geometric information on surface, we extract the fine-grained
surface-aligned features $\mathbf{f}_{\mathcal{S}}$ of a query point
$\mathbf{X}$ as the weighted sum of the associated features
$\mathbf{F}_{\mathcal{S}}$ of its $K$ nearest neighbors in a point cloud
$\mathcal{S}$, which is reconstructed from the input image $\mathbf{I}$ using
a point set generator $G_{\mathcal{S}}$.

Thus, our PVSeRF is conditioned on $\mathbf{f_{I}}$, $\mathbf{f_{V}}$, and
$\mathbf{f}_{\mathcal{S}}$ and can be reformulated as:

$\sigma,\mathbf{r}=\mathrm{PVSeRF}(\mathbf{X},\mathbf{d};\mathbf{f_{I}}\oplus\mathbf{f_{V}}\oplus\mathbf{f}_{\mathcal{S}})$
(2)

where $\oplus$ denotes a concatenation operation. Thanks to the incorporation
of $\mathbf{f_{V}}$ and $\mathbf{f}_{\mathcal{S}}$, the previously ambiguous
points that share the same $\mathbf{f_{I}}$ are now separable by the
concatenation
$\mathbf{f_{I}}\oplus\mathbf{f_{V}}\oplus\mathbf{f}_{\mathcal{S}}$. We present
more details about each component of our method as follows.

###  3.2 Feature Extraction

Pixel-aligned Features Following pixelNeRF [66], we also use pixel-aligned
features that contain fine-grained details about the scene’s geometry and
appearance properties to learn neural radiance fields. Given an input image
$\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, we employ a fully-convolutional
image encoder $E$ implemented by ResNet-34 [16] to extract its multi-scale
feature maps
$\\{\mathbf{F_{I}^{0}},\mathbf{F_{I}^{1}},\mathbf{F_{I}^{2}},\mathbf{F_{I}^{3}}\\}$,
which are the intermediate features at ’conv1’, ’layer1’, ’layer2’, and
’layer3’ of ResNet-34 but upsampled to the size of the input image
$\mathbf{I}$. Then, we acquire the pixel-aligned feature vector
$\mathbf{f_{I}}$ of a query 3D point $\mathbf{X}$ by projecting $\mathbf{X}$
to the 2D image coordinates $x$, and bilinearly interpolating the feature maps
concatenated by
$\\{\mathbf{F_{I}^{0}},\mathbf{F_{I}^{1}},\mathbf{F_{I}^{2}},\mathbf{F_{I}^{3}}\\}$
through $\mathcal{B}$:

$\mathbf{f_{I}}=\mathcal{B}(\mathbf{F_{I}^{0}}\oplus\mathbf{F_{I}^{1}}\oplus\mathbf{F_{I}^{2}}\oplus\mathbf{F_{I}^{3}},\mathbf{K}\mathbf{R}\mathbf{X})$
(3)

where $\oplus$ represents feature concatenation. However, $\mathbf{KR}$ may
project multiple 3D points $\mathbf{X}$ along $\mathbf{R}$ to a single
position on the 2D image coordinates, leading to ambiguous $\mathbf{f_{I}}$
and blurry synthesized novel views. To clarify such ambiguity, we propose to
augment $\mathbf{f_{I}}$ with complementary geometric features, including both
coarse voxel-aligned features learned from a volumetric grid, and fine
surface-aligned features extracted from a regressed point cloud.

Voxel-aligned Features We compute the voxel-aligned feature $\mathbf{f_{V}}$
with respect to $\mathbf{X}$ as follows. First, we reconstruct a volumetric
feature grid $\mathbf{F_{V}}\in\mathbb{R}^{32\times 32\times 32\times C}$ from
the input image $\mathbf{I}$ using a volume generator consisting of a VGG-16
[44] image encoder and a 3D CNN decoder. Then, we have:

$\mathbf{f_{V}}=\mathcal{T}(\mathbf{F_{V}},\Omega(\mathbf{X}))$ (4)

where $\mathcal{T}$ is a multi-scale trilinear interpolation inspired by
GeoPiFu [17] and IFNet [5], $\Omega(\mathbf{X})$ is a point set around
$\mathbf{X}$:

$\Omega(\mathbf{X})=\\{\mathbf{X}+s\cdot\mathbf{n}|\mathbf{n}=(1,0,0),(0,1,0),(0,0,1),...\\}$
(5)

where $s\in\mathbb{R}$ is the step length, $\mathbf{n}\in\mathbb{R}^{3}$
represents the unit vectors defined along the three axes in a Cartesian
coordinate system. Intuitively, $\mathbf{f_{V}}$ is a concatenation of all
queried feature vectors at points in $\Omega(\mathbf{X})$ that are trilinearly
interpolated from $\mathbf{F_{V}}$.

Surface-aligned Features Although they capture a global context about the
shape of a 3D object, voxel-aligned features are queried from a low-resolution
volumetric grid and thus lack geometric information on surface. As a
complement, we introduce surface-aligned features that capture fine details of
surface to facilitate radiance field learning. Given an input image
$\mathbf{I}$, we first regress a sparse point cloud $\mathcal{S}$ of size 1024
from $\mathbf{I}$ using a point set generator $G_{\mathcal{S}}$ based on
GraphX-convolutions [32]. Then, we feed the generated point cloud to a
PointNet++ [42] network to extract point-wise features
$\mathbf{F}_{\mathcal{S}}$. For each query point $\mathbf{X}$, we define its
surface-aligned feature $\mathbf{f}_{\mathcal{S}}$ as the weighted sum of the
corresponding feature vectors of $\mathbf{X}$’s $K$-nearest neighbors in
$\mathcal{S}$:

$\mathbf{f}_{\mathcal{S}}=\sum_{k=0}^{K}w_{k}*\mathbf{F}_{\mathcal{S}_{m(k)}}$
(6)

where $m(k),k=0,1,2...,K$ is the indices of the $K$ points, $w_{k}$ is
inversely proportional to its distance to $\mathbf{X}$:

$w_{k}=1/(1+\exp(||\mathbf{X}-\mathcal{S}_{m(k)}||)$ (7)

In this way, the features from the nearest neighbor contributes most to the
$\mathbf{f}_{\mathcal{S}}$, and vice versa.

###  3.3 Radiance Field Prediction and Rendering

Given an input single-view image $\mathbf{I}$, we construct its radiance field
with a MLP $f$, which regresses the volume density $\mathbf{\sigma}$ and view-
dependent radiance $\mathbf{r}$ from the 3D coordinates of a query point
$\mathbf{X}$, a viewing direction $\mathbf{d}$, and the corresponding pixel-,
voxel-, and surface-aligned features (  _i.e_. $\mathbf{f_{I}}$,
$\mathbf{f_{V}}$, and $\mathbf{f_{S}}$) of $\mathbf{I}$:

$\sigma,\mathbf{r}=f(\gamma_{m}(\mathbf{X}),\gamma_{n}(\mathbf{d});\mathbf{f_{I}}\oplus\mathbf{f_{V}}\oplus\mathbf{f_{S}})$
(8)

where $\gamma_{m}$ and $\gamma_{n}$ are position encoding functions [56, 29]
applied to $\mathbf{X}$, $\mathbf{d}$ respectively, which alleviates the
positional bias inherent in Cartesian coordinates without sacrificing their
discrepancy in-between. Specifically, $\gamma$ maps Cartesian coordinates from
$\mathbb{R}$ into a high dimensional space $\mathbb{R}^{2L}$:

$\displaystyle\gamma_{L}(\mathbf{p})=(\sin(2^{0}\pi\mathbf{p}),$
$\displaystyle\cos(2^{0}\pi\mathbf{p}),$ (9)
$\displaystyle...,\sin(2^{L-1}\pi\mathbf{p}),\cos(2^{L-1}\pi\mathbf{p}))$

where $\gamma(\cdot)$ is applied separately to each component of vector
$\mathbf{p}$. With the constructed radiance field represented by $\sigma$ and
$\mathbf{r}$, we render novel view images via differentiable rendering as:

$\mathbf{c_{t}}=\sum_{i}\tau_{i}(1-\exp(-\sigma_{i}))\mathbf{r_{i}}$ (10)

where $\mathbf{c_{t}}$ is the rendered pixel color,
$\tau_{i}=\exp(-\sum_{j=1}^{i-1}\sigma_{j})$ denotes the volume transmittance.
Intuitively, Eq. 10 shows that the a pixel’s radiance value (_i.e_. RGB color)
can be calculated by casting a ray from the camera to the pixel and
accumulating the radiance of points sampled on the ray.

###  3.4 Loss Functions

Corresponding to our pixel-, voxel- and surface-aligned features, we train our
model using three different loss functions as follows.

RGB Rendering Loss Similar to existing works in the NeRF series, we use $L2$
rendering loss as the main loss function. It constrains that the rendered
color value of each ray should be consistent with the corresponding ground-
truth pixel value. Thus, we have:

$L_{r}=||\mathbf{c_{t}}-\mathbf{\hat{c_{t}}}||_{2}^{2}$ (11)

where $\mathbf{c_{t}}$ and $\mathbf{\hat{c_{t}}}$ are the predicted and
ground-truth color values of sampled pixels from novel view $\mathbf{I_{t}}$
with viewpoint $\mathbf{R_{t}}$ respectively.

Volume Reconstruction Loss To learn volumetric features $\mathbf{F_{V}}$, we
add a 3D convolutional layer after $\mathbf{F_{V}}$ to estimate a low-
resolution occupancy volume $\mathbf{V}\in\mathbb{R}^{32\times 32\times 32}$,
whose ground-truth label is $\mathbf{V}^{*}$. Then, we apply a standard binary
cross-entropy loss and have:

$L_{v}=\sum_{i\in[1:32]^{3}}\mathbf{V}^{*}(i)\log\mathbf{V}(i)+(1-\mathbf{V}^{*}(i))\log(1-\mathbf{V}(i))$
(12)

Point Regression Loss We employ the Chamfer distance to constraint our point
set generation and have:

$L_{p}=\sum_{\mathbf{q}\in\mathcal{S}}\min_{\mathbf{q}^{*}\in\mathcal{S}^{*}}\|\mathbf{q}-\mathbf{q}^{*}\|^{2}\
\
+\sum_{\mathbf{q}^{*}\in\mathcal{S}^{*}}\min_{\mathbf{q}\in\mathcal{S}}\|\mathbf{q}-\mathbf{q}^{*}\|^{2}$
(13)

where $\mathcal{S}$ is the predicted point set and $\mathcal{S}^{*}$ is its
corresponding ground truth.

Overall Loss Function Our overall loss function is:

$L=\lambda_{1}*L_{r}+\lambda_{2}*L_{v}+\lambda_{3}*L_{p}$ (14)

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are weighting
parameters.

##  4 Experiments

|  | plane | bench | cbnt. | car | chair | disp. | lamp | spkr. | rifle | sofa | table | phone | boat | mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| DVR [35] | 25.29 | 22.64 | 24.47 | 23.95 | 19.91 | 20.86 | 23.27 | 20.78 | 23.44 | 23.35 | 21.53 | 24.18 | 25.09 | 22.70
$\uparrow$ PSNR | SRN [46] | 26.62 | 22.20 | 23.42 | 24.40 | 21.85 | 19.07 | 22.17 | 21.04 | 24.95 | 23.65 | 22.45 | 20.87 | 25.86 | 23.28
| pixelNeRF [66] | 29.76 | 26.35 | 27.72 | 27.58 | 23.84 | 24.22 | 28.58 | 24.44 | 30.60 | 26.94 | 25.59 | 27.13 | 29.18 | 26.80
| Ours | 31.32 | 27.43 | 28.40 | 28.12 | 24.37 | 24.61 | 28.73 | 24.44 | 30.82 | 27.42 | 26.60 | 26.99 | 29.92 | 27.48
| DVR [35] | 0.905 | 0.866 | 0.877 | 0.909 | 0.787 | 0.814 | 0.849 | 0.798 | 0.916 | 0.868 | 0.840 | 0.892 | 0.902 | 0.860
$\uparrow$ SSIM | SRN [46] | 0.901 | 0.837 | 0.831 | 0.897 | 0.814 | 0.744 | 0.801 | 0.779 | 0.913 | 0.851 | 0.828 | 0.811 | 0.898 | 0.849
| pixelNeRF [66] | 0.947 | 0.911 | 0.910 | 0.942 | 0.858 | 0.867 | 0.913 | 0.855 | 0.968 | 0.908 | 0.898 | 0.922 | 0.939 | 0.910
| Ours | 0.956 | 0.923 | 0.912 | 0.940 | 0.869 | 0.867 | 0.915 | 0.853 | 0.965 | 0.912 | 0.911 | 0.915 | 0.940 | 0.915
| DVR [35] | 0.095 | 0.129 | 0.125 | 0.098 | 0.173 | 0.150 | 0.172 | 0.170 | 0.094 | 0.119 | 0.139 | 0.110 | 0.116 | 0.130
$\downarrow$ LPIPS | SRN [46] | 0.111 | 0.150 | 0.147 | 0.115 | 0.152 | 0.197 | 0.210 | 0.178 | 0.111 | 0.129 | 0.135 | 0.165 | 0.134 | 0.139
| pixelNeRF [66] | 0.084 | 0.116 | 0.105 | 0.095 | 0.146 | 0.129 | 0.114 | 0.141 | 0.066 | 0.116 | 0.098 | 0.097 | 0.111 | 0.108
| Ours | 0.065 | 0.098 | 0.097 | 0.087 | 0.128 | 0.133 | 0.104 | 0.140 | 0.066 | 0.104 | 0.082 | 0.107 | 0.101 | 0.096

Table 1: Quantitative comparison on category-agnostic view synthesis. We color
code each row as best and second best. Our method outperforms all baselines by
a wide margin in terms of all mean metrics. Figure 4: Qualitative comparison
on category-agnostic view synthesis. A single model is trained among 13
ShapeNet categories, and tested on a single image for novel view synthesis. We
observe that our method produces detailed novel views, and is consistent in
both geometry and appearance. Conversely, pixelNeRF [66] fails to infer
correct geometry and produce inconsistent and blurry textures.

To demonstrate the superiority of our PVSeRF, we first compare it against
state-of-the-art methods on two single-image novel view synthesis tasks,
_i.e_. category-agnostic view synthesis and category-specific view synthesis.
Then, we evaluate our approach on real images, demonstrating the
generalization ability of our method. Finally, we conduct ablation studies to
validate the effectiveness of each component of our PVSeRF.

Datasets We benchmark our method extensively on the synthetic images from the
ShapeNet [2] dataset. Specifically, for the category-agnostic view synthesis
task, we use the renderings and splits from Kato _et al_. [19] which renders
objects from 13 categories of the ShapeNetCore-V1 dataset. Each object was
rendered at 64$\times$64 resolution from 24 equidistant azimuth angles, with a
fixed elevation angle. For the category-specific view synthesis task, we use
the dataset and splits provided by Sitzmann _et al_. [46], which renders 6,591
chairs and 3,514 cars from the ShapeNetCore V2 dataset. For the evaluation on
real images, we use the collected real-world cars images from [23]. To provide
supervision for volume reconstruction and point set regression, we convert
each ground-truth mesh to a point set of size $2048$ and a volumetric grid of
resolution $32^{3}$.

#### Implementation Details

We implement our model with PyTorch [40]. Details of the network architecture
are presented in the supplementary material. The training process of our
approach consists of two stages: i) we pre-train the volume generator
$G_{\mathbf{V}}$ and the point set generator $G_{\mathbf{S}}$ respectively
using loss functions defined in Eq. 12 and Eq. 13. Specifically,
$G_{\mathbf{V}}$ is trained with an initial learning rate of $10^{-3}$ and a
batch size of $64$ for $250$ epochs. The learning rate drops by a factor of
$5$ after $150$ epochs. $G_{\mathcal{S}}$ is trained with an initial learning
rate of $10^{-5}$ and a batch size of $4$ for $10$ epochs. The learning rate
drops by a factor of $3$ after $5$ and $8$ epochs. ii) we fine-tune the whole
network for $400$ epochs. We set the learning rate as $10^{-4}$ and the batch
size as $4$. We use an Adam [20] optimizer for all the training mentioned
above. We empirically set hyper-parameters $s=0.0722$, $K=5$, $m=6$, $n=0$,
$\lambda_{1}=\lambda_{2}=\lambda_{3}=1$.

#### Evaluation Protocol

Following the community standards [29, 46, 35], we use peak signal-to-noise
ratio (PSNR) and structural similarity index (SSIM) [58] to measure the
quality of the synthesized novel views. We also use LPIPS [67] that has been
shown to be closer to human perception.

###  4.1 Category-agnostic View Synthesis

Category-agnostic novel view synthesis aims to learn object priors that can
generalize across multiple categories.

#### Baselines

We compare our method against three closely-related state-of-the-art methods:
SRN [46], DVR [35] and pixelNeRF [66], which are applicable to synthesize
novel views for all categories. For DVR and pixelNeRF, we use pretrained
models from their official Github repositories111Niemeyer  _et al_. [35]:
https://github.com/autonomousvision/differentiable
_volumetric_rendering, Yu  _et al_. [66]: https://github.com/sxyu/pixel-nerf..
For SRN [46], we use the model trained by [66] to make it comparable with [46,
66]. All methods are trained using the same dataset and settings introduced in
Fig. 4. To facilitate a fair comparison, we follow the random view indices
provided by pixelNeRF and select the input view for each test object
accordingly.

#### Results

As Fig. 4 shows, our method outperforms all previous methods by synthesizing
more detailed novel views. In addition, it can be observed that i) the two
baseline methods, DVR [35] and SRN [46], tend to generate blurry images and
distorted geometries; ii) pixelNeRF [66] shows blurry and inconsistent
appearance. The quantitative results in Table 1 further justify the
superiority of our method against all baselines in terms of the mean values of
PSNR, SSIM and LPIPS metrics. Notably, the PSNR of our approach attains a
significant improvement over the second best method by $0.68$.

###  4.2 Category-specific View Synthesis

For category-specific view synthesis, all methods are trained on the chair or
car categories of ShapeNet [2].

#### Baselines

We choose SRN [45] and pixelNeRF [66] as the baseline methods. We also report
the quantitative results from TCO [50] and dGQN [9] provided by [46], to keep
in line with prior arts.

Figure 5: Qualitative comparison on category-specific view synthesis. The
performance of our method is comparable to that of the state-of-the-art
pixelNeRF [66]. Figure 6: Failure case of explicit geometry reasoning. Under
the challenging viewpoint, the scene geometry is ambiguously captured in a
single image, causing the network being unable to predict plausible
geometries.

#### Results

We show the quantitative and qualitative results in Table 2 and Fig. 5
respectively. It can be observed that the performance of our method is
comparable to the state-of-the-art method [66] both qualitatively and
quantitatively. Such comparable results indicate that the advantages of our
method are not significant in some neural rendering cases. We carefully
investigate the results and ascribe this to the invalidity of explicit
geometry reasoning in some cases (Fig. 6). Since the renderings provided by
[46] contain many challenging camera viewpoints, the explicit geometry
reasoning from single-view becomes a more challenging problem. We postpone the
discussion of this phenomenon to Sec. 5.

Figure 7: Novel view synthesis results on real car images. Although
extensively trained on synthetic data, our method can easily generalize to
real single-view images, and produce plausible view synthesis results and
underlying geometries.

###  4.3 Novel View Synthesis on Real Images

To highlight the generalization ability of our method, we evaluate our
pretrained models directly on real images without any finetuning.
Specifically, we first take the images from the Stanford cars dataset [23] and
apply the PointRend model [21] to mask their clutter backgrounds. Then, we
feed the preprocessed images into a category-specific model of ShapeNet “cars”
to predict novel views. As Fig. 7 shows, our method can not only synthesize
visually compelling novel views, but also infer accurate geometries. This
effectively demonstrates the excellent generalization performance of our
method on real image as it is only trained on synthetic images.

|  | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---
| TCO [50] | 21.27 | 0.88
| dGQN [9] | 21.59 | 0.87
Chairs | SRN [46] | 22.89 | 0.89
| pixelNeRF [66] | 23.72 | 0.91
| Ours | 23.33 | 0.91
| SRN [46] | 22.25 | 0.89
Cars | pixelNeRF [66] | 23.17 | 0.90
| Ours | 22.98 | 0.90

Table 2: Quantitative comparison on category-specific view synthesis. Since
the renderings from [46] contain many challenging camera viewpoints, our
performance is degenerated ascribe to the invalidity of explicit geometry
reasoning. Nevertheless, our method is comparable to the state-of-the-art
method [66]. Figure 8: Illustration of the complementary properties of point
set and volumes. We randomly show several predicted geometries. It can be seen
that these two representations exhibit reciprocal behaviors: the missing parts
of volumetric grid are spanned by point set, while the regions where the point
set is too sparse are occupied by volumes.

|  | plane | bench | cbnt. | car | chair | disp. | lamp | spkr. | rifle | sofa | table | phone | boat | mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| w/o joint | 29.03 | 25.18 | 26.12 | 25.82 | 21.97 | 22.25 | 26.33 | 22.19 | 28.55 | 25.18 | 24.27 | 24.67 | 27.54 | 25.16
| w/o surface-aligned | 30.82 | 27.00 | 28.31 | 27.67 | 24.05 | 24.33 | 28.73 | 24.33 | 30.63 | 26.97 | 26.27 | 26.85 | 29.58 | 27.15
$\uparrow$ PSNR | w/o voxel-aligned | 30.83 | 27.14 | 28.40 | 27.93 | 24.35 | 24.66 | 29.10 | 24.75 | 31.05 | 27.29 | 26.48 | 27.01 | 29.61 | 27.38
| Ours | 31.32 | 27.43 | 28.40 | 28.12 | 24.37 | 24.61 | 28.73 | 24.44 | 30.82 | 27.42 | 26.60 | 26.99 | 29.92 | 27.48
| w/o joint | 0.926 | 0.863 | 0.864 | 0.924 | 0.844 | 0.803 | 0.826 | 0.812 | 0.947 | 0.878 | 0.853 | 0.836 | 0.923 | 0.876
| w/o surface-aligned | 0.954 | 0.917 | 0.912 | 0.936 | 0.860 | 0.860 | 0.915 | 0.849 | 0.967 | 0.904 | 0.906 | 0.912 | 0.940 | 0.911
$\uparrow$ SSIM | w/o voxel-aligned | 0.955 | 0.921 | 0.913 | 0.941 | 0.867 | 0.870 | 0.915 | 0.856 | 0.966 | 0.911 | 0.910 | 0.917 | 0.939 | 0.915
| Ours | 0.956 | 0.923 | 0.912 | 0.940 | 0.869 | 0.867 | 0.915 | 0.853 | 0.965 | 0.912 | 0.911 | 0.915 | 0.940 | 0.915
| w/o joint | 0.097 | 0.134 | 0.134 | 0.099 | 0.137 | 0.163 | 0.175 | 0.164 | 0.098 | 0.123 | 0.120 | 0.149 | 0.120 | 0.123
| w/o surface-aligned | 0.064 | 0.100 | 0.096 | 0.094 | 0.136 | 0.132 | 0.102 | 0.144 | 0.064 | 0.108 | 0.085 | 0.100 | 0.100 | 0.099
$\downarrow$ LPIPS | w/o voxel-aligned | 0.063 | 0.096 | 0.096 | 0.085 | 0.130 | 0.126 | 0.100 | 0.137 | 0.060 | 0.104 | 0.081 | 0.104 | 0.100 | 0.095
| Ours | 0.065 | 0.098 | 0.097 | 0.087 | 0.128 | 0.133 | 0.104 | 0.140 | 0.066 | 0.104 | 0.082 | 0.107 | 0.101 | 0.096

Table 3: Quantitative comparison of ablation studies. Our joint method that
employs complementary coarse volumetric features and fine surface features
achieves the best performance. Whereas, removing any part of the proposed
method will cause more or less deterioration.

###  4.4 Ablation Study

To validate the effectiveness of each proposed component, we conduct an
ablation study on our method, yielding three variants: i) w/o surface-aligned
feature, in which only pixel- and voxel-aligned features are incorporated; ii)
w/o voxel-aligned feature, where the radiance field is conditioned only on
pixel- and surface-aligned features; iii) w/o joint training, in which we fix
all feature extractors222We use a PointNet++ [42] model trained on PartNet
[30] segmentation task as a point-feature extractor. and solely train the
radiance field predictor $f$. As Table 3 shows, it can be observed that the
w/o joint training variant constantly performs the worst among all variants.
This demonstrates that the joint learning of pixel-, voxel- and surface-
aligned features is crucial in our explicit geometric reasoning. In addition,
the performance of the w/o surface-aligned variant is always worse than the
w/o voxel-aligned feature variant, as the voxel-aligned features queried from
a low-resolution volume are better at capturing global geometry contexts. Our
full method achieves the best performance among all variants, which validates
the effectiveness of employing a hybrid of geometric features that complement
each other. This is also demonstrated in Fig. 8.

##  5 Conclusion

For the task of novel view synthesis from single-view RGB images, we present
PVSeRF, a novel learning framework that reconstructs neural radiance fields
conditioned on joint pixel-, voxel-, and surface-aligned features. By
augmenting hybrid geometric features with image features, we effectively
address the feature confusion issue of pixel-aligned features. Compare to
previous arts, our framework gains superior or comparable results in terms of
both visual perception and quantitative measures. Moreover, a suite of
ablation studies also verify the efficacy of our key contributions.

#### Limitation and Future Works

Despite the effectiveness of our method, there are still some limitations to
be addressed in future work. First, the performance of our method is dependent
on the amount of geometric information within the input single-view image. As
discussed in Sec. 4.2, when the scene geometry is little captured in the input
image due to challenging viewpoints, the novel views synthesized by our method
may become less clear. In future work, we plan to include multi-view
consistency as an additional supervision to train our geometry reasoning
network, thereby increasing its robustness to challenging viewpoints.
Secondly, we focus on the geometry reasoning from complete geometries (_i.e_.
surface and voxel) of 3D shapes that reconstructs neural radiance fields from
single-view RGB image and have not investigated that from more challenging
partial geometries (_e.g_. depth maps or multiplane images [55, 60]). In
future work, we plan to extend our method to such partial geometries, thereby
making our method more flexible.

## References

  * [1] Neill DF Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla.  Using multiple hypotheses to improve depth-maps for multi-view stereo.  In European Conference on Computer Vision, pages 766–779. Springer, 2008.
  * [2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [3] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  ArXiv, abs/2103.15595, 2021.
  * [4] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In CVPR, 2019.
  * [5] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.  Implicit functions in feature space for 3d shape reconstruction and completion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6970–6981, 2020.
  * [6] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll.  Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7911–7920, 2021.
  * [7] Christopher Bongsoo Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In ECCV, 2016.
  * [8] Abe Davis, Marc Levoy, and Fredo Durand.  Unstructured light fields.  In Computer Graphics Forum, volume 31, pages 305–314. Wiley Online Library, 2012.
  * [9] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al.  Neural scene representation and rendering.  Science, 360(6394):1204–1210, 2018.
  * [10] Haoqiang Fan, Hao Su, and Leonidas J. Guibas.  A point set generation network for 3d object reconstruction from a single image.  In CVPR, 2017.
  * [11] Jiahao Geng, Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou.  Warp-guided gans for single-photo facial animation.  ACM Transactions on Graphics (TOG), 37:1 – 12, 2018.
  * [12] Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen.  The lumigraph.  In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 43–54, 1996.
  * [13] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry.  Atlasnet: A papier-mâché approach to learning 3d surface generation.  In CVPR, 2018.
  * [14] Xiaoguang Han, Zhen Li, Haibin Huang, Evangelos Kalogerakis, and Yizhou Yu.  High-resolution shape completion using deep neural networks for global structure and local geometry inference.  In ICCV, 2017.
  * [15] Richard Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  2000\.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto.  Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction.  arXiv preprint arXiv:2006.08072, 2020.
  * [18] Heiko Hirschmuller.  Stereo processing by semiglobal matching and mutual information.  IEEE Transactions on pattern analysis and machine intelligence, 30(2):328–341, 2007.
  * [19] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3d mesh renderer.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3907–3916, 2018.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross B. Girshick.  Pointrend: Image segmentation as rendering.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9796–9805, 2020.
  * [22] Jean Kossaifi, Linh Hoang Tran, Yannis Panagakis, and Maja Pantic.  Gagan: Geometry-aware generative adversarial networks.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 878–887, 2018.
  * [23] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [24] Marc Levoy and Pat Hanrahan.  Light field rendering.  In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31–42, 1996.
  * [25] David B. Lindell, Julien N. P. Martel, and Gordon Wetzstein.  Autoint: Automatic integration for fast neural volume rendering.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14551–14560, 2021.
  * [26] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  ArXiv, abs/2007.11571, 2020.
  * [27] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7206–7215, 2021.
  * [28] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, 2019.
  * [29] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [30] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su.  PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [31] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty Reddy Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger.  Donerf: Towards real‐time rendering of compact neural radiance fields using depth oracle networks.  Computer Graphics Forum, 40, 2021.
  * [32] Anh-Duc Nguyen, Seonghwa Choi, Woojae Kim, and Sanghoon Lee.  Graphx-convolution for point cloud deformation in 2d-to-3d conversion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8628–8637, 2019.
  * [33] Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang.  Rendernet: A deep convolutional network for differentiable rendering from 3d shapes.  In NeurIPS, 2018.
  * [34] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yongliang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7587–7596, 2019.
  * [35] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504–3515, 2020.
  * [36] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia.  Deep mesh reconstruction from single rgb images via topology modification networks.  In ICCV, 2019.
  * [37] Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, and Alexander C Berg.  Transformation-grounded image generation network for novel 3d view synthesis.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 3500–3509, 2017.
  * [38] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [39] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  * [40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [41] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In ECCV, 2020.
  * [42] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In NeurIPS, 2017.
  * [43] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  arXiv preprint arXiv:2103.13744, 2021.
  * [44] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [45] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019.
  * [46] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  arXiv preprint arXiv:1906.01618, 2019.
  * [47] Jiapeng Tang, Xiaoguang Han, Junyi Pan, Kui Jia, and Xin Tong.  A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images.  In CVPR, 2019.
  * [48] Jiapeng Tang, Xiaoguang Han, Mingkui Tan, Xin Tong, and Kui Jia.  Skeletonnet: A topology-preserving solution for learning mesh reconstruction of object surfaces from rgb images.  arXiv preprint arXiv:2008.05742, 2020.
  * [49] Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang.  Sa-convonet: Sign-agnostic optimization of convolutional occupancy networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6504–6513, 2021.
  * [50] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Single-view to multi-view: Reconstructing unseen views with a convolutional network.  ArXiv, abs/1511.06702, 2015.
  * [51] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Multi-view 3d models from single images with a convolutional network.  In European Conference on Computer Vision, pages 322–337. Springer, 2016.
  * [52] Engin Tola, Christoph Strecha, and Pascal Fua.  Efficient large-scale multi-view stereo for ultra high-resolution image sets.  Machine Vision and Applications, 23(5):903–920, 2012.
  * [53] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a deforming scene from monocular video.  https://arxiv.org/abs/2012.12247, 2020.
  * [54] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d scene representation and rendering.  arXiv preprint arXiv:2010.04595, 2020.
  * [55] Richard Tucker and Noah Snavely.  Single-view view synthesis with multiplane images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 551–560, 2020.
  * [56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [57] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  * [58] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [59] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7467–7477, 2020.
  * [60] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8534–8543, 2021.
  * [61] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  In NeurIPS, 2016.
  * [62] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  * [63] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian.  Foldingnet: Point cloud auto-encoder via deep grid deformation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 206–215, 2018.
  * [64] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  arXiv preprint arXiv:2003.09852, 2020.
  * [65] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  arXiv preprint arXiv:2103.14024, 2021.
  * [66] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [67] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [68] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros.  View synthesis by appearance flow.  In European conference on computer vision, pages 286–301. Springer, 2016.
  * [69] Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, and Bill Freeman.  Visual object networks: Image generation with disentangled 3d representations.  In NeurIPS, 2018.


# Veni, Vidi, Vici: Solving the Myriad of Challenges before Knowledge Graph
Learning

1st Jeffrey Sardina  School of Computer Science and Statistics
Trinity College Dublin and Accenture Labs
Dublin, Ireland
0000-0003-0654-2938  2nd Luca Costabello  BioInnovation Labs
Accenture
Dublin, Ireland
0000-0002-0720-9347  3rd Christophe Guéret  BioInnovation Labs
Accenture
Dublin, Ireland
0000-0002-8914-6107

###### Abstract

Knowledge Graphs (KGs) have become increasingly common for representing large-
scale linked data. However, their immense size has required graph learning
systems to assist humans in analysis, interpretation, and pattern detection.
While there have been promising results for researcher- and clinician-
empowerment through a variety of KG learning systems, we identify four key
deficiencies in state-of-the-art graph learning that simultaneously limit KG
learning performance and diminish the ability of humans to interface optimally
with these learning systems. These deficiencies are: 1) lack of expert
knowledge integration, 2) instability to node degree extremity in the KG, 3)
lack of consideration for uncertainty and relevance while learning, and 4)
lack of explainability. Furthermore, we characterise state-of-the-art attempts
to solve each of these problems and note that each attempt has largely been
isolated from attempts to solve the other problems. Through a formalisation of
these problems and a review of the literature that addresses them, we adopt
the position that not only are deficiencies in these four key areas holding
back human-KG empowerment, but that the divide-and-conquer approach to solving
these problems as individual units rather than a whole is a significant
barrier to the interface between humans and KG learning systems. We propose
that it is only through integrated, holistic solutions to the limitations of
KG learning systems that human and KG learning co-empowerment will be
efficiently affected. We finally present our ”Veni, Vidi, Vici” framework that
sets a roadmap for effectively and efficiently shifting to a holistic co-
empowerment model in both the KG learning and the broader machine learning
domain.

###### Index Terms:

Knowledge Graphs, Knowledge Graph Embedding, Relational Learning, Neuro-
Symbolic Learning, GNNs

##  I Introduction

Knowledge Graphs (KGs) are semantic data stores that model data as a set of
nodes and the relations between them [1]. The atomic unit of knowledge in a
knowledge graph is the triple, which consists of a single labelled head node,
a labelled tail node, and a directed, labelled edge that relates the head to
the tail. For example, the fact ”Sauron created the One Ring” could be written
as (Sauron, created, One-Ring).

The graph structure of KGs lends them very naturally to a variety of
intrinsically networked datasets, such as social networks, computer networked
systems, and biomedical drug-gene interaction networks. However, KG data has
grown to a size that precludes human analysis without computational
assistance. For example, Hetionet, a common benchmark dataset for drug
repurposing in bioinformatics, contains 2.25M triples – far more than a human
can analyse alone [2]. Other large KGs can range from 1M to 21M triples [3,
4].

In order to empower human use of these massive KGs, various KG learning
systems have been proposed to detect patterns and predict new information in
the domain of a given KG. Knowledge Graph Embeddings (KGEs) use machine
learning to attempt to represent the semantics of a KG in vector space [1, 5,
6]. Other KG learning approaches include logical and rule-based methods [7, 8,
9], path-based methods [7, 10], and Graph Neural Networks (GNNs) [11, 12, 13,
14].

KG learning systems have made significant progress in assisting with research
tasks, such as drug re-purposing [7, 15, 15] and drug-drug interaction
prediction [16, 17]. However, recent literature has identified some major
shortcomings of the current state-of-the-art approaches to KG learning
systems.

The first shortcoming is that most existing KG learning systems do not account
for the logical structure of the data, which means that these systems
typically cannot use an expert-curated ontologies even if they are available
[1, 5, 6, 12, 18]. Moreover, application of KGEs to real-world problems,
notably including high-potential-impact uses such as medical drug repurposing
and predictions about drugs and diseases in the biomedical context, are mostly
done on methods that make no attempt to model the logical structure of the
data [19, 20, 21, 22]. This means that many logical and causal inferences the
humans care about most are not well accounted for, either in theory or in
practice.

Moreover, KG learning systems have been shown to suffer from biases due to
varying distributions of node degrees: high-degree nodes are generally learned
much more reliably and with much greater embedding quality than low-degree
nodes across various KG learning methods [23, 24, 25]. Despite their generally
superior embedding quality, high-degree nodes can also have detrimental
effects on KG learning systems [15, 23], to the point that removing high-
degree domain-relevant data from the KG can boost performance [15].

KG learning systems also often lack the focus that is desired by human
researchers and clinicians: while in many cases KG learning systems are only
used for predicting certain types of facts in a graph, those models are
trained to predict all types of facts with equal emphasis on each one [15].
Most widely-used KG learning systems do not calculate or use uncertainty or
relevance scores for triples, which further prevents the model from focusing
on the most relevant and most reliable information [1, 5, 6, 26].

Finally, most KG learning systems were not created or evaluated for
explainability [5, 6, 25, 7, 10, 12, 24, 27]. Some logical methods do allow
explainability [7, 8, 28]; however, with few exceptions, these logical
approaches generally cannot handle large KGs [7]. Moreover, there is no
agreed-upon definition or concrete metric for explainability for KG learning
systems, which makes comparison of progress in this area a challenge in itself
[27, 29]. As a result, not only are KG learning systems opaque to humans, but
critically detecting the presence and cause of the other 3 problems is
severely hindered by the black-box nature of most KG learning systems.

Individually, these problems are each a major setback for human-AI empowerment
and advancements in the realm of KGs. Taken together, they present a bleak
outlook for the use of KG learning techniques and suggest an urgent need for
corrective innovation and development within the field.

In this position paper, we first formalise definitions and nomenclatures for
each of these four problems: lack of expert knowledge integration, instability
to extreme topological variation, inability to perform focused learning, and
lack of explainability. We present the details of each problem and give a
review of state-of-the-art techniques that attempt to address each one. Our
analysis leads us to the conclusion that the state-of-the-art is deficient not
only because of a general lack of robust answers to these problems, but
because existing solutions tend to address only one of these problems while
ignoring the others. We conclude by presenting our ”Veni, Vidi, Vici”
framework for how to represent and consider diverse problems in the same
conceptual space to best empower humans with robust, interpretable, and
reliable learning systems. Finally, we propose that use of the ”Veni, Vidi,
Vici” model will allow researchers to not only greatly advance KG learning
systems past these major challenges, but also fundamentally strengthen the
machine learning-human interface in many different domains.

##  II Expert Knowledge Integration

We first define expert knowledge as so: expert knowledge is knowledge about
the logic, dependency patterns, pragmatics, inferences, and analytic
approaches of a specific domain. From this, we formally define expert
knowledge integration as follows: expert knowledge integration is the act of
making a machine learning model explicitly aware of expert knowledge in how it
models tasks and/or data. While in different domains this will have widely
different manifestations, in the domain of KG learning systems it can be
presented in a very straight forward manner. We can say, without loss of
generality, that a KG learning system expresses expert knowledge integration
if and only if it is able to recognise different information with the exact
same graph topology.

For example, take Figure 1. To a human, it is obvious that if Gimli is a
friend of Legolas and Legolas is an enemy of Sauron, that Gimli should also be
an enemy of Sauron. On the other hand, there is no reason to assume a direct
connection between Gandalf and Mithril Armour based on the information given
in the left-hand graph. The two graphs are topologically identical, but very
distinct in information content.

Figure 1: Two graphs with identical topology but distinct information.

Modern KGEs, such as TransE, ComplEx, DistMult, RotatE, and ConvE [5, 6, 9],
as well as rule mining systems such as AMIE+ [28], GNNs [12, 13, 14], and
path-based KG learning systems [10] cannot take this expert information into
account and would learn these two graphs identically. Using this example as a
guide, we formalise the problem of expert knowledge integration as so: By what
methods is it possible to teach a KG learning system to distinguish knowledge
with identical topology but variable semantics? We call methods that attempt
to distinguish such graphs ”Expert Knowledge Methods”.

Note: although rule-mining and query-based approaches are at times discussed
as if they contained expert knowledge, rule-based systems such as AMIE+ and
IterE [28, 30] and query-answering methods that allow logical operations in
queries (such as Query2box and CQD) [31, 32] are not Expert Knowledge Systems.
We exclude them because none of them can distinguish the two graphs in Figure
1; i.e., their reasoning derives entirely from topological patterns, not from
domain expertise or expert knowledge.

###  II-A Expert Knowledge Methods

Modern literature has principally focused on two main methods for creating
expert-knowledge aware systems: use of ontological rules [8, 9, 18, 33, 34,
35] and use of non-ontological domain expertise [7, 15].

Ontology-based methods incorporate background knowledge expressed as
ontological rules at various stages of the learning pipeline. For example,
Alshahrani et al. use an ontology to complete an existing KG before attempting
to learn on it, meaning that all rules inferred by the ontology are explicitly
present in the graph [33]. In contrast, DL-Learner mines rules from an KG and
uses them to refine or expand an existing ontology, taking the initial form of
that ontology into account [8]. Injecting ontologies into training is also
done, either to create negative examples to learn from [18] or to explicitly
model for logical relationships during training [34, 35]. A more complete
analysis of the stages of ontology integration and ontology-guided learning
can be found in [9].

A second, more recent approach to integrating expert knowledge is using ”meta-
paths” procured by domain experts to guide learning along specific parts of a
KG. Meta-paths are paths of entities and relationships in a KG that are
identified not by the identity of nodes along them, but by their broader type.
For example, (Gandalf, friends-with, Aragorn) would have the meta-path
(Person, friends-with, Person). These meta-paths allow expects to create rules
that graph learners can use to bias their training. For example, PoLo uses
expert-curated meta-paths to guide a path-based reinforcement learner on KGs
[7]. Ratajczak et al. use expert-curated meta-paths to prune out information
from a graph that experts would not consider relevant for specific learning
tasks [15].

##  III Handling Extreme Topological Variation

The second issue facing KG learning systems to-date is that of their inability
to handle extremities in node degree and topological variation. We formally
define this problem as so: handling topological variation means capturing the
semantics of a node or edge equally regardless of its local connectivity
patterns. An example of this can be seen in Figure 2. Both graphs on the right
contain identical semantic information. However, the topology of the graphs
vary, which means that the degree of each node is greatly reduced in the
right-hand graph as opposed to the left-hand one.

Figure 2: Two graphs with distinct topology but identical information.

While the fact that Gandalf is an enemy of Sauron is implicit in the graph and
obvious to a human, it is not labelled explicitly in the graph. As the
remainder of this section will show, this means that traditional KG learning
methods will be much less apt to learn that Gandalf and Sauron are enemies.
Using this example as a guide, we formalise the problem of handling
topological variation as so: By what methods is it possible to teach a KG
learning system to robustly and consistently learn knowledge with distinct
topology but identical semantics? We call methods that attempt to distinguish
such graphs ”connectivity-tolerant methods”.

###  III-A Connectivity-Tolerant Methods

Topological imbalance in KGs has a variety of negative effects on learning
using traditional KGE models, GNNs, and path-based KG learning systems [15,
23, 24, 25]. Low-degree nodes embed at a much lower quality relative to high-
degree nodes [24], and high-degree nodes are sometimes predicted as answers
during inference simply because of their higher degree, not because of domain
relevance [23]. So-called ”super-hubs”, or nodes with extremely high degree in
the graph, also dilute information and hinder learning; this occurs even when
those nodes are highly relevant to the given domain [15, 23].

Existing solutions to topological imbalance include the work of Liu et al.,
which found that using meta-learning to make low-degree node embeddings more
similar to high-degree node embeddings improved performance of KGEs on low-
degree nodes [24]. The authors replicated this in GNNs with similar results
[25]. A study by Tang et al. found that explicitly modelling for node degree
in the GNN allowed the model to more robustly learn embeddings for low-degree
nodes [11]. However, even with these systems the core problem of poor
representations for low-degree node remains at best mitigated, not solved,
since modern KG learning paradigms rely principally on (a plurality of) local
connections for learning [11, 24, 25].

##  IV Focusing Learning

The third issue facing KG learning systems is that, even though many KG
learning-based tasks only have one type of prediction in mind, they attempt to
learn all predictive tasks with equal strength [15]. Moreover, all triples in
the input graph are generally considered equally relevant and true [5, 6].
This means that KG learning models cannot estimate of how certain or relevant
a fact is, nor use human-known uncertainty in how it models KG data. We
formally define the problem of modelling uncertainty as so: focused learning
is learning that explicitly models which triples are less certain or less
relevant during training.

We split this into two (somewhat overlapping) cases: dealing with uncertainty
and dealing with relevance. For example, take Figure 3. On the left, we have a
graph where scores are given representing how uncertain (near 0) or certain
(near 1) a fact is. On the right, another graph contains triples with no
annotations, but where it is possible that some connections are uncertain. For
instance, If Frodo is friends with Samwise, and Samwise dislikes Gollum, we
may be less certain that Gollum is friends with Frodo. Similarly, we might not
care about predicting friendship, but about predicting who travels with whom.
This would make only the Travels-with relationship directly relevant; other
relationships would be of use only to the extend that they assist learning to
predict triples about friendship.

Figure 3: Graph representing cases where we may care about uncertainty and
relevance of various triples.

Using this example as a guide, we formalise the problem of explicitly
modelling for uncertainty as so: By what methods is it possible to distinguish
certainty and relevance of triples within a KG? We call methods that attempt
to model uncertainty / relevance ”focused learning methods”.

###  IV-A Focused Learning Methods

One method for focused learning in the state-of-the-art is that taken by UKGE
and FocusE for KGEs: to use uncertainty / relevance information given as
additional triple-level labels in the KG, as is shown in the left of Figure 3
[26, 36]. Specifically, UKGE uses a set of out-of-band logical rules to model
how uncertainties interact, and thus only works with uncertainty [36]. FocusE
takes a more broad approach: it directly uses numeric labels to modify the
scoring layer of a KGE model with fewer assumptions, allowing it to model both
uncertainty and relevance equally well [26].

Uncertainty can also be modelled implicitly, in the absence of certainty or
relevance labels in the graph [11]. One method for this is used in SL-DSGCN, a
GNN method that models uncertainty using a Bayesian-based teacher-student
model [11]. The Bayesian Neural Network is trained not only to teach a student
network, but to also give uncertainty scores as it teaches [11]. These scores
are not input to the model, but learned and created automatically by the
Bayesian Network during training [11].

Finally, using expert-curated meta-path information can also help focus
learning on those specific paths [7, 15]. Two recent applications are
filtering methods that use meta-paths to select for only task-relevant
information as a pre-processing step [15] and the PoLo model, which uses
reinforcement learning and logical rules to explicitly reward learning done on
the given meta-paths.

##  V Explainability

For our purposes, we define explainability in keeping with what Lipton calls
”post-hoc interpretability”, or the ability to explain why a model made a
certain prediction [37]. Similarly, we say a method is explainable by design
if it was created to natively give such explanations. Under this definition of
explainability, almost all modern KGEs, GNNs, and path-based methods would not
be explainable by design since their predictions do not provide any method for
post-hoc explanation [5, 6, 10, 12].

Figure 4: A triple predicted to be true (light green) and false (dark red).
Explanation aims to distinguish for what reasons one is labelled as true and
the other as false.

Suppose we have a trained KG learning system that performs link prediction. As
illustrated in Figure 4, it will attempt to predict if a triple is true (light
green) or false (dark red); or in other words, distinguish why one graph
structure is predicted to exist while the other is not. This leads us to
formalise the problem of explanation of KG learning systems as so: By what
methods can we extract the reason(s) that a KG learning system predicted the
presence of a certain graph structure or feature rather than any other? We
call systems that implement these methods ”Explainability Systems”.

###  V-A Explainability Systems

The first general approach to explainability is to provide a post-hoc
explanation of a KG learning system that is not explainable by design, such as
TransE [5], RDF2vec [10], or SL-DSGCN [11]. Methods for post-hoc
interpretability of KGEs are generally based on estimating the influence of a
triple on a given prediction [38]. This includes various Instance Attribution
Methods such as Influence Functions and Instance and Gradient Similarity
computation [39]. For GNNs, many post-hoc explanation systems are not GNN-
specific [40, 41], although some, such as GNNExplainer, PGExplainer, and
GraphMask explicitly take advantage of the triple-based structure of graphs to
identify the most important triples for a prediction [40]. More detailed
analyses can be found in [38, 40, 41].

The second approach is to create models that are explainable by design. This
includes rule-based models such as AMIE+ and DL-Learner, which use symbolic
logic to infer rules that are used to predict new statements in a graph [28,
8]. Since those rules are written in human-readable logical clauses, they are
direct explanations for the models’ predictions. Models such as PoLo also fall
in this category: it uses expert-curated rules (in the form of meta-paths) to
create a policy-based model able to predict new triples [7].

##  VI Veni, Vidi, Vici: Towards a Common Conceptual Space

In examining the four major challenges faced by modern KG learning systems, we
observe that, while many papers attempt to solve each problem individually,
none attempt to solve all four at once. The current state-of-the-art has
divide-and-conquer methodology that results in tunnel-vision with respect to
specific problems with KG learning systems and a lack of attention to the
broader systemic shortcomings of the field as a whole. To this, we propose a
new design methodology, called ”Veni, Vidi, Vici” for helping human
researchers to conceptualise, frame, and solve distinct problems as one. While
we present this in the context of KG learning systems, it is potentially
applicable to any case where multiple challenges exist within the same broader
domain. An overview of this model and its use is shown in Figure 5.

Figure 5: An overview of the ”Veni, Vidi, Vici” framework.

The ”Veni, Vidi, Vici” approach occurs in three phases. The first, ”Veni”
(meaning ”I came” in Latin) is to phrase all problems in a shared conceptual
space. The second, ”Vidi” (meaning ”I saw” in Latin) is to deeply explore and
analyse this shared conceptual space: for example, what information is lost
when it splits into various sub-problems, and what elements of this space do
human users care most about? The third and final, ”Vici” (meaning ”I
conquered’ in Latin) is to determine how to solve the underlying problem
represented by this shared conceptual space as a single, atomic unit

As an example of the use of the ”Veni, Vidi, Vici” system, consider the four
challenges to KG learning systems presented here. The first step, Veni, we did
by taking each of these challenges and placing them in the same shared
conceptual space of distinguishing elements of graphs. Our Vidi step was to
examine each of these problems in the state-of-the-art and their limitations,
especially in terms of opportunity lost by treating each problem as distinct.
Finally, our Vici step of creating, evaluating, and publishing a joint
solution is left as a future direction – one that this methodology well
prepares up to take.

We expand on these three phases below.

###  VI-A Phase 1: Veni

The Veni phase is characterised by asking, how? Specifically, it asks How can
I understand these problems in a shared conceptual space? It aims to express
each of the four issues – expert knowledge integration, wide topological
variation, attention to certainty and relevance, and explainable – into the
same terms.

The idea here is twofold. First, that considering the problems in isolation
will lead to less general and less generalisable solutions to advancing KGE
systems, as outlined in the review above. The second is that, since there is a
single common goal (improving the predictive performance of KGE systems), that
the various barriers to this goal should be expressible in common, unifying
terms. The Veni step thus would not ask ”To what extent can some KGE method
reduce the detrimental effects of topological extremity in a KG?” – such a
question, while valuable, only focusses on one part of the problem, and does
not take a fully holistic approach.

Instead, it begins by looking for common ground in these problems. The review
above highlights that the methods that seek to strongly and meaningfully
integrate expert knowledge for inference on a graph (such as Polo) tend to
provide ways of estimating relevance and reducing the effect of extreme
topological variation [7]. We also note that most logic-based methods tend to
be more natively explainable than those that do not directly consider logic
[28, 8, 5, 6].

As a result, one possible question in the Veni phase would be the following:
”To what extent can path-based models of graph logic represent the relevant
semantic content of a KG?” Here, the common term chosen is logic – most
commonly supplied by an ontology for KGEs. It then asks how a logic based
method can represent content that is relevant and semantic, meaning that it
should be able to distinguish different semantics in identical structures, and
focus learning on the most important parts thereof. In other words, this is
all about distinguishing graph elements, as outlined in the previous section –
distinguishing the meaning of regions with same structure (logic and
semantics), distinguishing the relevance of distinct triples, and identifying
the structures of a graph that actually encode similar information (logic and
topological awareness).

Once this core question is obtained, the Veni step is complete – each problem
has been presented in terms of the same basic building-blocks.

###  VI-B Phase 2: Vidi

The Vidi phase is characterised by asking why?. Specifically, it asks two
questions Why did these problems originally separate? and How can I see what
information was lost in that separation?. The purpose of this phase is to
solidify understanding of the question asked in the Veni phase and,
critically, to ensure that the common terms being used to represent the
problems in the same conceptual space actually address the core issue of why
those problems should be modelled as one, rather than as individual units.

The review conducted here provides a hypothesis to answer these questions for
KGEs: that these problems originally separated because of the distinct
deficiencies identified by different researchers, and because of the differing
aims and purposes for which various groups developed and used KGEs. Similarly,
the main loss was that each of these problems was solved for a very specific
purpose, and that therefore general advancement of KGEs was second to
advancement in the specific areas most pressing at the time the research was
conducted. In other words, there remains a large region of untapped potential
for KGE development in providing more general and generalisable solutions to
KG learning.

###  VI-C Phase 3: Vici

The Vici phase is characterised by asking what? Specifically, it asks What
goals can I set to holistically address the patterns I see? This is the action
step – the step where the goals set are directly actionable. The output here
is a specific model or set of models, evaluation protocols, and tests to
ensure that any proposed solution not only answers the main question asked in
the Veni phase, but that it actually improves on each of the distinct problems
identified and recovers what was lost when each problem was taken separately
(as identified in the Vidi phase).

###  VI-D Bringing it all Together

The purpose of the Veni, Vidi, Vici model is to invite a new paradigm of
developing KGEs – to focus on a holistic, broad goal rather than a variety of
distinct sub-goals. Based on the existing literature and the review conducted
here, it is in the opinion of the authors that a holistic approach to
furthering KGE development will provide a novel, and potentially very
successful, new method for approaching semantic modelling and KG
representation.

It is critical to note that the proposed ”Veni, Vidi, Vici” is not a linear
model – it is expected that each phase will involve multiple steps of
backtracking to the previous (or first) stage for iterative refinement. It
also is meant to be used in any case where several challenges exist within a
single broader domain, and has potential to drive innovation in many fields
outside of KG learning systems, although exploration of how it can be applied
in other domains is left as a future direction.

##  VII Conclusion

In this paper, we provide a survey of the state-of-the-art in KG learning
systems and the challenges they face. We conclude that the current ”divide and
conquer” approach to advancing individual problems facing KG learning systems
is actively detrimental to advancing KG learning systems. To address this, we
propose the ”Veni, Vidi, Vici” system that allows human experts to reconsider
these problems in a common framework. This framework explicitly models human
desiderata and aims to help guide development along the directions that human
experts desire most. It simultaneously aims to ensure co-empowerment not only
of all areas of learning systems, but also how humans can interface with those
systems. Finally, we note that ”Veni, Vidi, Vici” is broadly applicable to any
domain with multiple sub-problems.

## References

  * [1] Hogan A, Blomqvist E, Cochez M, D’amato C, Melo GD, Gutierrez C, et al.  Knowledge Graphs.  ACM Comput Surv. 2021 jul;54(4).  Available from: https://doi.org/10.1145/3447772.
  * [2] Himmelstein DS, Lizee A, Hessler C, Brueggeman L, Chen SL, Hadley D, et al.  Systematic integration of biomedical knowledge prioritizes drugs for repurposing.  eLife. 2017 sep;6:e26726.  Available from: https://doi.org/10.7554/eLife.26726.
  * [3] Ali M, Berrendorf M, Hoyt CT, Vermue L, Galkin M, Sharifzadeh S, et al.  Bringing Light Into the Dark: A Large-Scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework.  IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022;44(12):8825-45.
  * [4] Speer R, Chin J, Havasi C.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.  In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. AAAI’17. AAAI Press; 2017. p. 4444–4451.
  * [5] Nickel M, Murphy K, Tresp V, Gabrilovich E.  A Review of Relational Machine Learning for Knowledge Graphs.  Proceedings of the IEEE. 2016;104(1):11-33.
  * [6] Wang Q, Mao Z, Wang B, Guo L.  Knowledge Graph Embedding: A Survey of Approaches and Applications.  IEEE Transactions on Knowledge and Data Engineering. 2017;29(12):2724-43.
  * [7] Liu Y, Hildebrandt M, Joblin M, Ringsquandl M, Raissouni R, Tresp V.  Neural Multi-Hop Reasoning With Logical Rules on Biomedical Knowledge Graphs.  In: Extended Semantic Web Conference; 2021. .
  * [8] Bühmann L, Lehmann J, Westphal P.  DL-Learner—A framework for inductive learning on the Semantic Web.  Journal of Web Semantics. 2016;39:15-24.  Available from: https://www.sciencedirect.com/science/article/pii/S157082681630018X.
  * [9] Zhang W, Chen J, Li J, Xu Z, Pan J, Chen H. Knowledge Graph Reasoning with Logics and Embeddings: Survey and Perspective; 2022.
  * [10] Ristoski P, Rosati J, Noia TD, Leone RD, Paulheim H.  RDF2Vec: RDF graph embeddings and their applications.  Semantic Web. 2019;10:721-52.
  * [11] Tang X, Yao H, Sun Y, Wang Y, Tang J, Aggarwal C, et al.  Investigating and Mitigating Degree-Related Biases in Graph Convoltuional Networks.  In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. CIKM ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 1435–1444.  Available from: https://doi.org/10.1145/3340531.3411872.
  * [12] Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z, et al.  Graph neural networks: A review of methods and applications.  AI Open. 2020;1:57-81.  Available from: https://www.sciencedirect.com/science/article/pii/S2666651021000012.
  * [13] Kipf T, Welling M.  Variational Graph Auto-Encoders.  ArXiv. 2016;abs/1611.07308.
  * [14] Jiang X, Zhu R, Li S, Ji P.  Co-embedding of Nodes and Edges with Graph Neural Networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020:1-1.
  * [15] Ratajczak F, Joblin M, Ringsquandl M, Hildebrandt M.  Task-driven knowledge graph filtering improves prioritizing drugs for repurposing.  BMC bioinformatics. 2022 March;23(1):84.  Available from: https://europepmc.org/articles/PMC8894843.
  * [16] Mohamed SK, Nounu A, Nováček V.  Biological applications of knowledge graph embedding models.  Briefings in Bioinformatics. 2020 02;22(2):1679-93.  Available from: https://doi.org/10.1093/bib/bbaa012.
  * [17] Celebi R, Uyar H, Yasar E, Gümüs O, Dikenelli O, Dumontier M.  Evaluation of knowledge graph embedding approaches for drug-drug interaction prediction in realistic settings.  BMC Bioinformatics. 2019 12;20.
  * [18] Jain N, Tran TK, Gad-Elrab M, Stepanova D.  In: Improving Knowledge Graph Embeddings with Ontological Reasoning; 2021\. p. 410-26.
  * [19] Celebi R, Uyar H, Yasar E, Gumus O, Dikenelli O, Dumontier M.  Evaluation of knowledge graph embedding approaches for drug-drug interaction prediction in realistic settings.  BMC bioinformatics. 2019;20(1):1-14.
  * [20] Sosa DN, Derry A, Guo M, Wei E, Brinton C, Altman RB.  A literature-based knowledge graph embedding method for identifying drug repurposing opportunities in rare diseases.  In: Pacific Symposium on Biocomputing 2020. World Scientific; 2019. p. 463-74.
  * [21] Mohamed SK, Nováček V, Nounu A.  Discovering protein drug targets using knowledge graph embeddings.  Bioinformatics. 2020;36(2):603-10.
  * [22] Mohamed SK, Nounu A, Nováček V.  Drug Target Discovery Using Knowledge Graph Embeddings.  In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing. SAC ’19. New York, NY, USA: Association for Computing Machinery; 2019\. p. 11–18.  Available from: https://doi.org/10.1145/3297280.3297282.
  * [23] Bonner S, Kirik U, Engkvist O, Tang J, Barrett IP.  Implications of topological imbalance for representation learning on biomedical knowledge graphs.  Briefings in Bioinformatics. 2022 07;23(5).  Bbac279.  Available from: https://doi.org/10.1093/bib/bbac279.
  * [24] Liu Z, Zhang W, Fang Y, Zhang X, Hoi SCH.  Towards Locality-Aware Meta-Learning of Tail Node Embeddings on Networks.  In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. CIKM ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 975–984.  Available from: https://doi.org/10.1145/3340531.3411910.
  * [25] Liu Z, Nguyen TK, Fang Y.  Tail-GNN: Tail-Node Graph Neural Networks.  In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. KDD ’21. New York, NY, USA: Association for Computing Machinery; 2021. p. 1109–1119.  Available from: https://doi.org/10.1145/3447548.3467276.
  * [26] Pai S, Costabello L.  Learning Embeddings from Knowledge Graphs With Numeric Edge Attributes.  In: Zhou ZH, editor. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. International Joint Conferences on Artificial Intelligence Organization; 2021. p. 2869-75.  Main Track.  Available from: https://doi.org/10.24963/ijcai.2021/395.
  * [27] Bianchi F, Rossiello G, Costabello L, Palmonari M, Minervini P. Knowledge Graph Embeddings and Explainable AI; 2020.
  * [28] Galárraga L, Teflioudi C, Hose K, Suchanek FM.  Fast Rule Mining in Ontological Knowledge Bases with AMIE+.  The VLDB Journal. 2015 dec;24(6):707–730.  Available from: https://doi.org/10.1007/s00778-015-0394-1.
  * [29] Doshi-Velez F, Kim B.  Towards A Rigorous Science of Interpretable Machine Learning.  arXiv: Machine Learning. 2017.
  * [30] Zhang W, Paudel B, Wang L, Chen J, Zhu H, Zhang W, et al.  Iteratively Learning Embeddings and Rules for Knowledge Graph Reasoning.  In: The World Wide Web Conference. WWW ’19. New York, NY, USA: Association for Computing Machinery; 2019. p. 2366–2377.  Available from: https://doi.org/10.1145/3308558.3313612.
  * [31] Ren* H, Hu* W, Leskovec J.  Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings.  In: International Conference on Learning Representations; 2020. Available from: https://openreview.net/forum?id=BJgr4kSFDS.
  * [32] Minervini P, Arakelyan E, Daza D, Cochez M.  Complex Query Answering with Neural Link Predictors (Extended Abstract)*.  In: Raedt LD, editor. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. International Joint Conferences on Artificial Intelligence Organization; 2022. p. 5309-13.  Sister Conferences Best Papers.  Available from: https://doi.org/10.24963/ijcai.2022/741.
  * [33] Alshahrani M, Khan MA, Maddouri O, Kinjo AR, Queralt-Rosinach N, Hoehndorf R.  Neuro-symbolic representation learning on biological knowledge graphs.  Bioinformatics. 2017 04;33(17):2723-30.  Available from: https://doi.org/10.1093/bioinformatics/btx275.
  * [34] Garg D, Ikbal S, Srivastava SK, Vishwakarma H, Karanam H, Subramaniam LV.  Quantum Embedding of Knowledge for Reasoning.  In: Wallach H, Larochelle H, Beygelzimer A, d'Alché-Buc F, Fox E, Garnett R, editors. Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc.; 2019. Available from: https://proceedings.neurips.cc/paper/2019/file/cb12d7f933e7d102c52231bf62b8a678-Paper.pdf.
  * [35] Kulmanov M, Liu-Wei W, Yan Y, Hoehndorf R.  El embeddings: Geometric construction of models for the description logic el++.  arXiv preprint arXiv:190210499. 2019.
  * [36] Chen X, Chen M, Shi W, Sun Y, Zaniolo C.  Embedding Uncertain Knowledge Graphs.  In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI’19/IAAI’19/EAAI’19. AAAI Press; 2019. Available from: https://doi.org/10.1609/aaai.v33i01.33013363.
  * [37] Lipton ZC.  The Mythos of Model Interpretability.  Commun ACM. 2018 sep;61(10):36–43.  Available from: https://doi.org/10.1145/3233231.
  * [38] Pezeshkpour P, Tian Y, Singh S.  Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications.  In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics; 2019. p. 3336-47.  Available from: https://aclanthology.org/N19-1337.
  * [39] Bhardwaj P, Kelleher J, Costabello L, O’Sullivan D.  Adversarial Attacks on Knowledge Graph Embeddings via Instance Attribution Methods.  In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics; 2021. p. 8225-39.  Available from: https://aclanthology.org/2021.emnlp-main.648.
  * [40] Li P, Yang Y, Pagnucco M, Song Y. Explainability in Graph Neural Networks: An Experimental Survey; 2022.
  * [41] Yuan H, Yu H, Gui S, Ji S. Explainability in Graph Neural Networks: A Taxonomic Survey; 2020.