# SO-NeRF: Active View Planning for NeRF using Surrogate Objectives

Keifer Lee1, Shubham Gupta1, Sunglyoung Kim1,2, Bhargav Makwana1,2,
Chao Chen1, Chen Feng1,
1New York University 2Denotes Equal Contribution
https://ai4ce.github.io/SO-NeRF The work is supported by NSF Awards 2238968
and 2322242. The corresponding author is Chen Feng<EMAIL_ADDRESS>
###### Abstract

Despite the great success of Neural Radiance Fields (NeRF), its data-gathering
process remains vague with only a general rule of thumb of “sampling as
densely as possible”. The lack of understanding of what actually constitutes
good views for NeRF makes it difficult to actively plan a sequence of views
that yield the maximal reconstruction quality. We propose Surrogate Objectives
for Active Radiance Fields (SOAR), which is a set of interpretable functions
that evaluates the goodness of views using geometric and photometric visual
cues - surface coverage, geometric complexity, textural complexity, and ray
diversity. Moreover, by learning to infer the SOAR scores from a deep network,
SOARNet, we are able to effectively select views in mere seconds instead of
hours, without the need for prior visits to all the candidate views or
training any radiance field during such planning. Our experiments show SOARNet
outperforms the baselines with $\sim$80x speed-up while achieving better or
comparable reconstruction qualities. We finally show that SOAR is model-
agnostic, thus it generalizes across fully neural-implicit to fully explicit
approaches.

##  1 Introduction

A key aspect of a practical and intelligent embodied agent is the ability to
dynamically respond and plan appropriately to an ever-changing environment -
dubbed active perception [2]. Of course, one presupposes the ability to
construct a reliable representation of the environment as a prior, and a class
of approach has been found to be particularly efficacious in recent years -
neural implicit representations with Neural Radiance Fields (NeRF) [29, 1].
Its applications ranges from Virtual Reality [44, 8, 26] to Simultaneous
Localization, Mapping (SLAM) [50, 38, 20].

However, in contrast to their popularity, the prototypical process of
constructing such representations remained vague with a general rule of thumb
- capture as many images of the subject as densely and directionally varied as
possible. The latter suggestion may be fine for one-off tasks or small
subjects, but grows to be intractable when either of the aforementioned
conditions are not satisfied; e.g. capturing images for thousands of products
or trying to reconstruct the Santa Maria del Fiore in its entirety for large-
scale exploration and neural topological mapping. In robotics context, a
properly perceived representation is key to planning and navigation.
Therefore, it is of interest to study more efficient ways of capturing the
required images, and indeed this is known as the Next Best View (NBV) problem
[34].

There have been many proposed approaches to tackle the NBV problem in the
classical Structure from Motion (SfM) setting [14, 16], and even in the recent
neural implicit framework [33, 21, 13, 35, 19, 45, 5]. However, these methods
so far suffer from some key problems from an application-viability perspective
in one way or another. Probabilistic uncertainty estimation methods such as
ActiveNeRF [33], NeurAR [35] and Bayes’ Rays [13] are either too slow or not
straightforward to work with by requiring training of a NeRF model -
effectively an in-flight or post hoc approach; they are also not interpretable
as abstract probabilistic uncertainties does not correspond to actual physical
qualities. On the other end of the spectrum, there are methods from classical
SfM tasks such as SCONE [14] that utilizes intuitive concepts like surface
coverage as a proxy objective to maximize for. However, these uni-dimensional
methods are too simplistic, which may not generalize well across object
classes with different structure.

Therefore, in this work, we introduce SOAR to tackle the aforementioned
problems of active perception for radiance fields models such as NeRF.
Specifically, the main contributions of our work are as follows:

  1. 1.

We propose a set of surrogate objectives, SOAR to characterize informative
samples for training radiance fields models. These functions correspond
directly to physical qualities, rendering them intuitive and explainable.

  2. 2.

Unlike in-flight and post hoc methods, SOAR is prescriptive, not needing any
radiance field model training at all. Therefore it is also radiance fields
model-agnostic.

  3. 3.

We further devised a method to efficiently compute these objectives with a
novel neural network, dubbed SOARNet, generating trajectories in seconds
instead of hours.

  4. 4.

Finally, SOAR provides an alternative training-free approach to estimating the
goodness of a radiance field model with respect to the input data. We will
release our code and data for reproducibility.

Figure 1: The overall pipeline of our proposed approach. (a) Given an object,
(b) we initialize via pseudo-coverage initialization to obtain a set of
starting poses $\mathcal{\hat{M}}$ and corresponding camera-poses
$\mathcal{\hat{C}}$. (c) We then generate a trajectory for sampling in a
greedy manner by computing the objective scores of all unseen camera-poses,
$c_{n+1}$ where $\mathcal{C}^{-}$ is the set of all unseen camera-poses. (e)
To compute said scores, we utilize our novel model, SOARNet for speedy
inference, and we then simply pick the candidate with the highest score,
$c_{z}$ as our next pose, $\hat{c}_{n+1}$ until we reach our budget,
$\mathcal{B}$ for (d) radiance fields training.

##  2 Related Work

Novel View Synthesis is generating views of the given object or scene from
previously unseen camera poses. This is a challenging task due to the high
uncertainty of elements that were previously occluded from the field of view
[46, 28, 27, 49]. While there are many approaches recently published [4, 22,
43, 47], we only explore Novel View Synthesis using radiance fields [29, 11]
to stay within the scope of this paper. Note that the original NeRF [29] by
Mildenhall et al. shall be referenced as Vanilla NeRF interchangeably
henceforth to distinct the generic idea of neural radiance field modeling and
the actual model itself.

The original NeRF while impressive was not very computationally efficient as
it encodes the object via a neural network (implicit representation). To
further improve the run-time performance, there have been recent works
alleviating the computation bottleneck [12], and these works can be
categorized as baked (fully-implicit), hybrid, and fully-explicit
representation methods [18, 30, 10, 6]. Fridovich-Keil et al. [10] voxelize
the scene and store the density and spherical harmonics coefficients as
scalars and vectors per color channel respectively. As such, slow neural
networks are not required, resulting in what’s known as a fully-explicit
representation. Müller et al. [30] leveraged a multi-resolution hash table of
feature vectors to lookup the pixel’s features, resulting in a massive speed-
up. Since their approach leverages a neural network for volumetric rendering,
such methods are classified as hybrid representation approaches. In the same
class, Chen et al. [6] proposed a method with massive speed-up and saving in
memory footprint compared to fully-explicit approaches by representing a scene
as a 4D tensor and decomposing the tensor into multiple low-rank components.

Next Best View planning is a highly relevant topic for robotic perception that
deals with finding the best sensor positioning to maximize information gain.
This also directly ties in with the idea of active perception [2, 3], where
the observer moves around a scene with an intention to better capture salient
information. Active perception has previously been employed for various tasks
[37, 31, 41] but only recently for training radiance fields model [33].

Guédon et al. [14] propose a hybrid method that predicts an occupancy grid so
as to maximize the total surface coverage using a neural networks and Monte
Carlo integration over a volumetric representation. Improving on that, [16]
utilizes only camera inputs and is able learn in a self-supervised fashion.
Kopanas and Drettakis [24] introduced observation frequency and angular
uniformity as metrics to improve convergence for free-viewpoint navigation.
Dhami et al. [9] propose a two-stage process that predicts the complete point
cloud using partial observations and selects for views that minimizes the
distance traveled.
Uncertainty estimation in NeRF has recently received a lot of interest. Pan et
al. [33] incorporates uncertainty estimation into the Vanilla NeRF training
pipeline, which learns to estimate the uncertainty of candidate poses in
parallel during training via Bayesian estimation. These candidates are then
greedily selected post-training to maximize the estimated information gain.
Lee et al. [25] proposes a ray-based uncertainty estimator by computing the
entropy of the distribution of the color samples along each ray. Similarly,
Shen et al. [39] presents a generalized NeRF model that can learn the
probability distribution of all radiance fields representing a given scene for
uncertainty estimation via Bayesian learning. Jin et al. [21] proposes a
generic framework that learns to predict the uncertainty of each candidate
poses, but without needing to train a radiance fields model. However, to
compute the uncertainties, their method still requires the images taken from
said candidate poses in the first place, which is a non-negligible limitation
in real-world applications.

Goli et al. [13] presents a post hoc framework to evaluate the uncertainty in
any pretrained NeRF model. The framework produces a spatial uncertainty field
by introducing perturbations along the learned camera rays and by utilizing
Bayesian Laplace approximations. Ran et al. [35] proposes a proxy for the Peak
Signal-to-Noise Ratio (PSNR) metric that is jointly trained with the
parameters of an implicit radiance fields model to predict a new quantity,
dubbed Neural Uncertainty for trajectory planning. Our approach, does not
perform post-hoc uncertainty estimation, but preemptively evaluates unseen
candidate poses via the SOAR scores.

##  3 Methodology

We begin by formulating the problem - given an object and a corresponding set
of images of said object,
$\mathcal{M}=\left\\{m_{1},m_{2},...,m_{N}\right\\}$, taken from camera-poses,
$\mathcal{C}=\left\\{c_{1},c_{2},...,c_{N}\right\\}$,
$c\in\mathbb{R}^{3\text{x}4}$ and a budget constraint,
$\mathcal{B}\in\mathbb{N}$ we want to select a subset of poses, subjected to
the constraint, that will maximize a reconstruction quality metric such as
PSNR, $\Gamma$ via a radiance fields model, $\Omega$. Formally,

$\hat{\mathcal{C}}=\operatorname*{arg\,max}_{c}\Gamma(\Omega(\mathcal{C},\mathcal{M}))\quad
s.t.\;\,\text{dim}(\hat{\mathcal{C}})=\mathcal{B}$ (1)

In the following subsections, we outline the different parts of the pipeline.
In 3.1, we formally define the objective functions of our approach from first
principles. In 3.2, we introduce SOARNet which enables us to efficiently
compute the objective scores. Lastly, in 3.3 we introduce the finalized
framework by delineating a simple trajectory planning schema with SOARNet.

###  3.1 Objective Functions Definitions

We begin with a set of presuppositions on the qualities of a training set that
would yield a NeRF with good reconstruction as measured via $\Gamma$.
Fundamentally, we hypothesized from first principles that a good training set
can be obtained by maximizing the surface coverage of the object, the
attention paid to local regions with particular geometric or textural
complexity, and the resulting multi-view diversity of observations.
Succinctly, we refer to the aforementioned qualities as the (i) Surface
Coverage, ${\mathbf{C}}$ (ii) Geometric Complexity, $\mathbf{Q}$ (iii)
Textural Complexity, $\mathbf{T}$ (iv) and Ray Diversity, $\mathbf{D}$
respectively.

Surface Coverage, $\mathbf{C}$ as defined is simply the ratio of seen faces,
${j}$ via the image taken from the $k$-th camera-pose, ${c_{k}}$, against the
total number of faces of the object or mesh, ${J}$. We can therefore define a
corresponding objective function for $\mathbf{C}$ as follows, where
$f_{C}\in\mathbb{R}$

$f_{C}=\frac{1}{J}\sum_{j}^{J}\mathrm{U}(\sum_{k}^{n}V_{j}^{k})$ (2)

where $\mathrm{U(\cdot)}$ is the Heaviside step function, $n$ is the total
number of camera-poses visited, and $V_{j}^{k}$ is an indicator function
corresponding to the visibility of face $j$ from the $k$-th camera-pose. Given
the mesh of an object, optimizing for Eq. (2) amounts to maximizing the number
of unique faces seen across the camera-poses.

Geometric Complexity, $\mathbf{Q}$ corresponds to the variability of the mesh
faces’ configurations, and one way to quantify this complexity is via the mesh
normals, $\hat{\mathrm{n}}$. For example, a geometrically complex mesh such as
a cathedral, would have a higher degree of surface normal variability compared
to a simple smooth ball due to the varied substructures comprising the
cathedral - e.g. pillars, the dome. In the case of the cathedral, we would
intuitively want to pay extra attention to the local regions that are complex
by sampling more varied views of said regions.

We formally define the objective function for $\mathbf{Q}$ as follows, where
$f_{Q}\in\mathbb{R}$

$f_{Q}^{\prime}=\left|\mathrm{LoG}(\hat{\mathrm{n}}_{j};\sigma_{Q})\right|\,\beta_{Q}\,V_{j}^{k}+1$
(3) $f_{Q}=\frac{1}{J(5\sigma_{Q})^{2}}\sum_{j}^{J}f_{Q}^{\prime}$ (4)

where $\mathrm{LoG(\cdot)}$ is the Laplacian of Gaussian filter, $\sigma_{Q}$
is the standard deviation of the Gaussian filter, $\beta_{Q}$ a scaling
factor, $f_{Q}^{\prime}$ is the per-face geometric complexity and $f_{Q}$ is
the normalized total complexity. We take the absolute value of the output
since the directionality of the normal variations does not matter.
Maximization of Eq. 4 will encourage the selection of camera-poses with higher
geometric complexity as defined above. In practice, a scaling discount factor,
$d_{Q}$ is applied towards the $\mathbf{Q}$ values of overlapping faces seen
during trajectory planning in order to control for over-emphasis on certain
local regions with unusually high geometric complexity, trapping it in a local
maxima.

Textural Complexity, $\mathbf{T}$ corresponds to the photometric variation of
images taken at camera-pose, $c_{k}$ of the object. This means that
$\mathbf{T}$ is computed with respect to the 2D projections of the object
instead of the underlying 3D mesh. We have chosen this simplified approach for
$\mathbf{T}$ since unlike geometric complexities which may be hard to discern
on 2D projections, textural characteristics are not. Similar to Geometric
Complexity, $\mathbf{Q}$ we hypothesized that regions with high textural
complexity should be paid extra attention. Formally, we define the
corresponding objective function for $\mathbf{T}$ as follows, where
$f_{T}\in\mathbb{R}$

$f_{T}^{\prime}=\mathrm{Hist}\left(\mathrm{LBP}\left(m_{k};r_{T},p_{T}\right)\right)$
(5) $f_{T}=1-f_{T}^{\prime}\left[b\right]$ (6)

where $\mathrm{Hist(\cdot)}$ is simply the histogram function, $b$ the last
bin’s index, $\mathrm{LBP(\cdot)}$ is the Local Binary Pattern texture
descriptor [32], $r_{T}$ and $p_{T}$ are the filter radius and the number of
circularly symmetric local neighborhood points of the descriptor, and finally
$m_{k}$ is the image taken at camera-pose, $c_{k}$. Similar to $f_{Q}$, in
practice, we also impose a discount factor, $d_{T}$ towards the $\mathbf{T}$
values seen during trajectory planning to prevent overfitting, and we used
$r_{T}$ of 3 and $p_{T}$ of 3. Finally, we compute the final score by
averaging $f_{T}$ computed on the Hue and Saturation channels of the image
respectively. Maximization of Eq. (6) is equivalent to selecting the image
with the highest textural complexity.

Figure 2: Architecture of the SOARNet.

Ray Diversity, $\mathbf{D}$ as defined is the variation of the rays, $i$
projected from various viewing poses observing a particular face, $j$ of the
object mesh. Intuitively, having highly varied novel views of the object faces
would yield better reconstruction. The idea is closely related to densely
sampling a set of training images from as many views as possible to improve
multi-view correspondences. We went a step further here by exactly quantifying
said idea with a formal definition of the objective function as follows

$\theta^{k}_{ij}=\left|\frac{180^{\circ}}{\pi}\arccos{\left(\frac{i\hat{\mathrm{n}}_{j}}{\left|i\right|\left|\hat{\mathrm{n}}_{j}\right|}\right)}\right|-90^{\circ}$
(7)

where $\theta^{k}_{ij}$ is the augmented ray-surface grazing angle between ray
$i$ from camera-pose $c_{k}$ and face $j$ with surface normal
$\hat{\mathrm{n}}_{j}$. However, as opposed to indiscriminately maximizing for
view-angle variance, we impose a penalty towards any ray observing a face
beyond a certain grazing angle via soft-thresholding. The intuition behind the
penalty is derived from the observation that it becomes increasingly difficult
to visually obtain useful information about a given object’s face as we
approach the orthogonal angle of its surface normal. We perform said soft-
thresholding by assigning each observing ray, $i$ with an outlier score

$\psi^{k}_{ij}=1-\frac{\alpha_{1}}{1+e^{-\theta^{k}_{ij}+\alpha_{2}}}$ (8)

where $\psi^{k}_{ij}$ is the outlier score, and $\alpha_{1}$ and $\alpha_{2}$
are hyperparameters to control the grazing angle soft-threshold with values of
$1$ and $-10$ respectively. We then compute the total outlier ratio of face
$j$ as follows

$\phi_{j}=\frac{1}{K_{j}}\sum_{k}^{n}\left(\sum_{i}^{I}\psi^{k}_{ij}V^{k}_{ij}\right)$
(9)

where

$K_{j}=\sum_{k}^{n}\sum_{i}^{I}V^{k}_{ij}$ (10)

which simplifies Eq. (9)

$\phi_{j}=\mathbb{E}\left[\psi^{k}_{ij}\right]$ (11)

Next, we compute the ray diversity, $d_{j}$ of face $j$ as follows

$\delta_{j}=\frac{1}{I^{3}}\left[\sum_{i}^{I}\left(x_{i}-\bar{x}_{i}\right)^{2}\sum_{i}^{I}\left(y_{i}-\bar{y}_{i}\right)^{2}\sum_{i}^{I}\left(z_{i}-\bar{z}_{i}\right)^{2}\right]$
(12)

where $x,y,z$ are the corresponding principal axis values of each observing
ray $i$. This is akin to multiplying the variances of each individual
principal axis of the observing rays.

$\delta_{j}=\sigma_{x}^{2}\sigma_{y}^{2}\sigma_{z}^{2}$ (13)

Finally, the normalized effective Ray Diversity objective function is as
follows, where $f_{D}\in\mathbb{R}$

$f_{D}=\frac{1}{J}\sum_{j}^{J}\delta_{j}\phi_{j}$ (14)

Note that for all the objective functions of $f_{C}$, $f_{Q}$, $f_{D}$, and
$f_{T}$, their values are further smoothened and clipped with the following
smoothing function, $\xi:\mathbb{R}\mapsto\left[0,1\right]$ with $\alpha_{3}$
of $3$.

$\xi(x)=\mathrm{max}\left(\mathrm{min}\left(\frac{1}{1+\left[e^{(-\alpha_{3}ex)+(\alpha_{3}0.5e)}\right]},1\right),0\right)$
(15)

###  3.2 Efficient Computation with SOARNet

With the objective functions defined, we now turn our attention to efficient
computation. By default, the ground truth mesh is required to compute the
objective functions, which limits its applicability. In order to remediate
this, we proposed a Deep Neural Network, SOARNet to infer the respective
objective scores for the poses visited so far,
$\mathcal{\hat{C}}=\left\\{\hat{c}_{k}:k\leq n\right\\}$ and a further
unvisited pose, $c_{n+1}$ given a set of corresponding monocular 2D images up
to step $n$, $\mathcal{\hat{M}}=\left\\{m_{k}:k\leq n\right\\}$, with
constraint $n+1\leq\mathcal{B}$, and the aforementioned camera parameters of
each image; note that our method does not require the image of the unvisited
camera-pose as input. Formally,

$\textbf{SOARNet}\left(\mathcal{\hat{C}},\mathcal{\hat{M}},c_{n+1}\right)=\left\\{\mathbf{F}_{g}:g\in\\{n,n+1\\}\right\\}$
(16)

where

$\mathbf{F}_{g}=\left\\{f_{C}^{g},f_{Q}^{g},f_{D}^{g},f_{T}^{g}\right\\}$ (17)

At each step, the elements of $\mathbf{F}_{n}$ correspond to the normalized
cumulative scores of all poses seen so far for the corresponding objectives.
Likewise, $\mathbf{F}_{n+1}$ returns the normalized cumulative scores from an
unseen camera-pose, $c_{n+1}$ conditioned upon the visited poses.

The architecture of the model can be found in Fig. 2, where $\mu(\cdot)$ is a
ResNet [17] encoder, $o(\cdot)$ is an MLP encoder taking positionally encoded
camera extrinsics, and $G(\cdot)$ and $H(\cdot)$ are simple MLP based
prediction heads for generating the corresponding objective scores,
$\mathbf{F}_{g}$ via regression. For loss functions, we used the Huber loss on
the predictions of each head and take their sum as the final loss,
$\mathcal{L}$ as follows

$\mathcal{L}=\mathcal{L}_{\text{G}}\left(\mathbf{\hat{F}}_{n},\mathbf{F}_{n}\right)+\mathcal{L}_{\text{H}}\left(\mathbf{\hat{F}}_{n+1},\mathbf{F}_{n+1}\right)$
(18)

Algorithm 1 Trajectory Planning

1:procedure GenerateTrajectory

2: $B\leftarrow\text{length of }\text{budget}$

3: $\mathcal{P}\leftarrow\text{objective sequence,
}[\mathbf{C},\mathbf{Q},\mathbf{Q},\mathbf{D},\mathbf{D},\mathbf{T}]$

4: $\mathcal{C}^{-}\leftarrow\text{set of unseen poses}$

5: $\mathcal{\hat{C}}\leftarrow\text{camera poses list}$

6: $\mathcal{\hat{M}}\leftarrow\text{images list}$

7: $n_{\text{init}}\leftarrow\text{number of initialization steps}$

8: while step $\leq B$ do

9: if step $\leq n_{\text{init}}$ then

10: do PseudoCoverage

11: else

12: $idx\leftarrow step\;\%\;\textit{len}(\mathcal{P})$

13: objective, $f$ $\leftarrow\mathcal{P}[idx]$

14: $scores\leftarrow\text{empty }array$

15: for $c_{z}$ in $\mathcal{C}^{-}$ do

16:
$s_{f}\xleftarrow[]{\text{select}f}\text{SOARNet}\left(\mathcal{\hat{C}},\mathcal{\hat{M}},c_{z}\right)$

17: $scores.append\left(s_{f}\right)$

18: $\mathcal{\hat{C}}.append\left(argmax(scores)\right)$

19: $update$ $\mathcal{\hat{M}}$

20: $step\leftarrow step+1$

21: done

###  3.3 Trajectory Generation with SOARNet

Once we are able to determine the importance of each candidate pose, the next
step would be constructing an optimal trajectory. We propose a simple yet
effective greedy strategy by picking the view that maximizes the desired
objective at each step. At once, another question naturally follows from the
proposed approach, which objective function should be optimized for? We
observed that the optimal objective function, is class-dependent - some object
classes are better addressed by surface coverage, $\mathbf{C}$ and some via
textural complexity, $\mathbf{T}$ instead. Intuitively, this makes sense since
each object may physically and visually differ significantly. Therefore, if we
simply maximize the total objective scores, it may not actually be picking the
candidate with the maximal marginal score of the most effective objective
function for the object.

Hence, our trajectory planning algorithm maximizes only a single objective per
step in an interleaved manner. This way, we are able to generate a trajectory
that accounts for all objectives as a whole. From our empirical testing, we
have found that $\mathbf{D}$ and $\mathbf{Q}$ to be most helpful, and
therefore we gave particular emphasis to these objectives in our ensemble with
the following sequence,
$[\mathbf{C},\mathbf{Q},\mathbf{Q},\mathbf{D},\mathbf{D},\mathbf{T}]$; for
brevity, we refer to this as the ensemble objective, $\mathbf{E}$.

Finally, we hypothesize and indeed observe that having good coverage is of
particular importance during initialization when available information is
minimal. As such, we utilize an effective yet easy-to-compute pseudo-coverage
initialization schema which simply picks poses at the extremum along each
principal spatial axis for $n_{\text{init}}$ number of poses to begin with.
Succinctly, the trajectory planning schema for $\mathbf{E}$ is summarized in
Algorithm 1 with $n_{\text{init}}=3$.

##  4 Experiment

In this section, we first present our experimental dataset and setup in Sec.
4.1, before delving more into the results with respect to various baselines in
Sec.4.2.

###  4.1 Dataset and Setup

|  PSNR ($\uparrow$) |  LPIPS ($\downarrow$) |  SSIM ($\uparrow$) |  Time ($\downarrow$)
---|---|---|---|---
Object | AN | Ours | AN | Ours | AN | Ours | AN (TG) | AN (RFT) | Ours (TG) | Ours (RFT)
drums | 17.92 | 18.25 | 0.332 | 0.322 | 0.779 | 0.774 | 14h 12m | 20s | 11m 42s
ficus | 21.55 | 21.81 | 0.169 | 0.163 | 0.869 | 0.877 | 14h 18m | 20s | 15m 6s
hotdog | 25.36 | 28.77 | 0.202 | 0.123 | 0.888 | 0.921 | 14h 10m | 20s | 10m 13s
lego | 23.57 | 25.17 | 0.169 | 0.127 | 0.835 | 0.865 | 14h 12m | 20s | 9m 46s
materials | 19.54 | 21.27 | 0.234 | 0.192 | 0.811 | 0.839 | 14h 22m | 20s | 10m 4s
mic | 24.66 | 24.94 | 0.146 | 0.135 | 0.913 | 0.918 | 14h 14m | 20s | 8m 36s
ship | 21.27 | 21.14 | 0.334 | 0.332 | 0.721 | 0.716 | 14h 17m | 20s | 11m 28s
Table 1: Reconstruction quality of TensoRF [6] trained via trajectory
generated ActiveNeRF, AN [33] and Ensemble (Ours) on NeRF Synthetic. For time,
TG is the trajectory generation time, and RFT is the radiance fields training
time. Best result is represented with bold.

Dataset. For our experiments, we have created a diverse dataset of 640 objects
across 20 classes based on the Objaverse [7] dataset containing Large
Vocabulary Instance Segmentation categories [15]. Similar to NeRF-Synthetic
[29], for each object we sample images taken from random poses on a fixed-size
spherical surface looking inward towards the object, with 100 images for
training, 100 for validation and 200 for testing using Blender. More
information on the generation process can be found in supplementary.

From this set of objects, we manually computed the individual $f_{C}$,
$f_{Q}$, $f_{D}$, and $f_{T}$ objectives’ score, and generated corresponding
trajectories of length $30$ via a random walk on the full train-set. Our final
train, validation, and test sets have approximately $31$k, $1.1$k, and $5$k
samples respectively. For the final evaluation with actual radiance fields
training, we further selected a subset of $50$ objects from the test set to do
so for computational feasibility; this subset will be referred to as the
final-set for brevity. Finally, we also leveraged the NeRF-Synthetic dataset
for out-of-distribution performance test.

Modelling Radiance Fields. For radiance fields modeling, we utilized TensoRF
[6] as our primary model. TensoRF adopts a hybrid approach to neural radiance
fields modeling unlike the Vanilla NeRF, which enables it to achieve
significant speed-up in model training and convergence via tensor
decomposition. However, since the proposed objective functions are agnostic
with respect to the radiance fields modeling method, we hypothesized that it
should be effective regardless of the variants. Indeed, we show that our
method generalizes across radiance fields model variants in Sec. 4.2.

Experimental Setup. The SOARNet model was trained on a machine with an NVIDIA
RTX8000 GPU, 64GB of DRAM and $8$ CPUs. The model was trained with a step-
scheduler of step size of $0.6$ and $\gamma$ of $0.5$, and the Adam optimizer
[23] at a learning rate of $3e-5$, $\beta_{1}$ of $0.9$ and $\beta_{2}$ of
$0.999$, $\epsilon$ of $1e-08$, with AMSGrad [36] enabled, and no weight
decay. Finally, we trained the model for $12$ epochs with a batch-size of $8$.

We leverage the NerfStudio [40] implementations of TensoRF [6] and Vanilla-
NeRF [29] for radiance field training. We also use Plenoxels [10] and
ActiveNeRF [33] provided by the respective authors. For details on the exact
configurations, please refer to supplementary.

###  4.2 Results Evaluation

Figure 3: Example renders with our method against the baseline, ActiveNeRF. Here we used a budget of $\mathcal{B}=30$ for all instances. Vanilla NeRF | Plenoxels
---|---
|  PSNR ($\uparrow$) |  LPIPS ($\downarrow$) |  SSIM ($\uparrow$) |  PSNR ($\uparrow$) |  LPIPS ($\downarrow$) |  SSIM ($\uparrow$)
Object | Rand | Ours | Rand | Ours | Rand | Ours | Rand | Ours | Rand | Ours | Rand | Ours
bookcase | 33.83 | 35.02 | 0.042 | 0.038 | 0.975 | 0.979 | 25.32 | 26.67 | 0.123 | 0.107 | 0.927 | 0.940
cone | 31.50 | 31.75 | 0.059 | 0.055 | 0.969 | 0.970 | – | – | – | – | – | –
earphone | 30.94 | 31.74 | 0.029 | 0.023 | 0.974 | 0.977 | 26.10 | 27.41 | 0.098 | 0.084 | 0.940 | 0.947
globe | 28.83 | 28.52 | 0.039 | 0.043 | 0.958 | 0.955 | 23.89 | 23.61 | 0.129 | 0.131 | 0.916 | 0.911
shoe | 26.31 | 26.82 | 0.054 | 0.050 | 0.950 | 0.952 | – | – | – | – | – | –
snowman | 35.63 | 36.93 | 0.017 | 0.014 | 0.991 | 0.993 | 31.87 | 33.24 | 0.049 | 0.050 | 0.977 | 0.978
Table 2: Reconstruction quality of Vanilla NeRF [29] and Plenoxels [10]
trained via trajectories generated from Ensemble (Ours) against the Random
baseline. Results here are based on a subset of 6 objects from our Objaverse
test-set and averaged across a range of budgets. The best result is
represented with bold.

Primary Evaluation. We first evaluate the performance of our model on the
NeRF-Synthetic dataset with unseen classes and objects of higher complexity
for a challenging out-of-distribution test. Note that chair has been removed
from the evaluation since our original training dataset contains said class.
The result can be found in Tab. 1. For this evaluation, we compare our
performance against a prior approach, ActiveNeRF [33]. Additionally, we have
also trained multiple instances of each object across a range of budget, with
$\mathcal{B}\in\left\\{3,6,12,20,30\right\\}$. This way, we should be able to
get a better representation of the potency of each method. We reported the
averaged result across said budgets on 3 reconstruction metrics of Peak
Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [42]
and Perceptual Similarity (LPIPS) [48].

Firstly, note that the reconstruction quality of a model trained via our
Ensemble schema, E outperforms ActiveNeRF in almost every class and across all
reconstruction quality metrics, lagging behind only on ship by a small margin,
demonstrating its potency. Secondly, since our proposed approach is model
agnostic for radiance fields modeling, we can therefore decouple trajectory
planning from the radiance fields training. Succinctly, this means, unlike
[33, 35] where they are locked into a particular radiance fields model, which
could be very slow to train, our approach does not impose such a restriction
and does not require any radiance fields training for trajectory planning. For
these reasons, we are able to achieve a significant speed-up, with an average
$\sim$80x improvement compared to the ActiveNeRF. In Tab. 1 we report the
total time taken as two separate components - the trajectory generation time,
TG and the radiance fields training time, RFT. For ActiveNeRF we report only a
single value for both time components since the two processes are inherently
one. Finally, we retrained additional TensoRF instances with the trajectories
generated via ActiveNeRF with the same configurations so the reconstruction
metrics would be comparable; for fairness, we have omitted the time taken for
said retraining. It should be noted that our approach does not need the image
taken from a candidate unseen pose for scoring, only the camera extrinsics,
rendering it much more useful in real-world applications (e.g. does not need
tedious sampling by actual visitation of poses). Renders of sample views of
the methods can be found in Fig. 3 for visual comparison.

For the in-distribution evaluation, we trained TensoRF on the custom Objaverse
final-set via 6 trajectory planning schemas - Random as a baseline, Ensemble
as described in Sec. 3.3, and the maximization of individual objective
functions ${f_{C}}$, $f_{Q}$, $f_{D}$ and $f_{D}$. For the random baseline, we
independently generated 5 trajectories per object and reported their averaged
result. As mentioned prior, the optimal objective function could differ across
objects based on their physical representation. As such, we have grouped the
set of results trained on the trajectory generated via the individual
objective functions and took the best performing amongst them as another
candidate schema dubbed, Optimal CQDT. The results can be found in Tab. 3; we
used a simple baseline here due to limited computational resource.

Method |  PSNR ($\uparrow$) |  LPIPS ($\downarrow$) |  SSIM ($\uparrow$)
---|---|---|---
Random | 29.07 | 0.081 | 0.951
E (Ours) | 29.45 | 0.071 | 0.955
OP (Ours) | 29.67 | 0.068 | 0.957
Table 3: Averaged result of Optimal CQDT, OP and Ensemble, E against the
Random baseline. The best result is represented with bold, and second best
with underline.

From the table, we can observe that Optimal CQDT performs the best, with
Ensemble coming in as the second best, outperforming the Random baselines. The
superior performance of Optimal CQDT reveals that the objects from the
Objavserse dataset can be saliently captured and represented by a single
objective function. However, it should also be noted that the efficacy may not
be homogeneous across the objective functions for a given object. For example,
for bookcase, $f_{D}$ is worse than $f_{C}$. We hypothesized that this is due
to the complexity of the shelving and large-flat surfaces, which creates a
local minima for $f_{D}$, resulting in an over concentration around a local
region. On the flip side, for soccer_ball, the random baseline is actually
competitive compared to the other schemas. We hypothesized that for objects
that are geometrically simple and easily covered from all directions, random
walk could suffice since the difference in saliency across poses is minimal.
From this observation, we further hypothesized that for such objects, one
could simply utilize a cheap random walk for trajectory planning. The
breakdown of the result by class can be found in the supplementary.

Lastly, Ensemble does indeed offer a viable alternative to trajectory planning
when the optimal objective function is not known ahead of time, or if it’s too
costly to be determined explicitly.
Model Variant Generalization. Intuitively, our approach should also generalize
across radiance field model variants since the objective functions are
grounded in geometry and photometric visual cues. Therefore, we also tested
for such generalization by evaluating the approach on a subset of our
Objaverse dataset with a fully neural-implicit radiance fields model, Vanilla
NeRF and a fully explicit model, Plenoxels. With these, our evaluations
covered the spectrum of radiance fields models from fully neural-implicit to
fully explicit models. The result of the evaluation can be found in Tab. 2.
For the results on cone and shoe with Plenoxels, we have omitted them as
Plenoxels is unable to converge for them at all. The result validates our
hypothesis by demonstrating that indeed, the proposed schema remain
efficacious across model variants in almost all cases.

##  5 Conclusion

Limitation. The current ensembling schema is quite simple and by no means
exhaustive; our future studies will consider more effective ways to merge the
proposed objective functions. Additionally, we currently focus only on inward-
looking scenes with singular objects, which is not representative of the
complex environment one may encounter in complex mapping and navigation tasks.
Therefore, we would also like to apply SOAR to more challenging large-scale
environments in future works.

Summary. In this work, we have proposed SOAR, a set of objective functions
designed to serve as surrogates to indicate the goodness of a resultant
radiance fields model, given a set of inputs. In order to enable efficient
inference for real-time trajectory generation, we further proposed a deep
neural network, SOARNet that enables per-step planning in $<$1s on unseen
poses. With extensive evaluations, we have shown that indeed our approach is
able to outperform various baselines, even in challenging out-of-distribution
scenarios and across radiance fields model variants.

Acknowledgement We extend our gratitude to several individuals and teams whose
support contributed to the fruition of our work. We express our appreciation
to Mohit Mehta and Anbang Yang for their helpful discussion, and to the NYU
High Performance Computing team for their support in our experimentation. The
work is supported by NSF Awards 2238968 and 2322242.

## References

  * Arandjelović and Zisserman [2021] Relja Arandjelović and Andrew Zisserman.  Nerf in detail: Learning to sample for view synthesis.  _arXiv preprint arXiv:2106.05264_ , 2021.
  * Bajcsy [1988] R. Bajcsy.  Active perception.  _Proceedings of the IEEE_ , 76(8):966–1005, 1988.
  * Bajcsy et al. [2018] Ruzena Bajcsy, Yiannis Aloimonos, and John K Tsotsos.  Revisiting active perception.  _Autonomous Robots_ , 42:177–196, 2018.
  * Chan et al. [2023] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.  Generative novel view synthesis with 3d-aware diffusion models.  _arXiv preprint arXiv:2304.02602_ , 2023.
  * Chen et al. [2021] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14124–14133, 2021.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Deitke et al. [2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13142–13153, 2023.
  * Deng et al. [2022] Nianchen Deng, Zhenyi He, Jiannan Ye, Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, and Qi Sun.  Fov-nerf: Foveated neural radiance fields for virtual reality.  _IEEE Transactions on Visualization and Computer Graphics_ , 28(11):3854–3864, 2022.
  * Dhami et al. [2023] Harnaik Dhami, Vishnu D Sharma, and Pratap Tokekar.  Pred-nbv: Prediction-guided next-best-view for 3d object reconstruction.  _arXiv preprint arXiv:2304.11465_ , 2023.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5501–5510, 2022.
  * Gao et al. [2022] Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, and Jonathan Li.  Nerf: Neural radiance field in 3d vision, a comprehensive review.  _arXiv preprint arXiv:2210.00379_ , 2022.
  * Garbin et al. [2021] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14346–14355, 2021.
  * Goli et al. [2023] Lily Goli, Cody Reading, Silvia Selllán, Alec Jacobson, and Andrea Tagliasacchi.  Bayes’ rays: Uncertainty quantification for neural radiance fields.  _arXiv preprint arXiv:2309.03185_ , 2023.
  * Guédon et al. [2022] Antoine Guédon, Pascal Monasse, and Vincent Lepetit.  Scone: Surface coverage optimization in unknown environments by volumetric integration.  _Advances in Neural Information Processing Systems_ , 35:20731–20743, 2022.
  * Gupta et al. [2019] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5356–5364, 2019.
  * Guédon et al. [2023] Antoine Guédon, Tom Monnier, Pascal Monasse, and Vincent Lepetit.  MACARONS: Mapping And Coverage Anticipation with RGB ONline Self-supervision.  In _CVPR_ , 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hedman et al. [2021] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec.  Baking neural radiance fields for real-time view synthesis.  _ICCV_ , 2021.
  * Jain et al. [2021] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5885–5894, 2021.
  * Jiang et al. [2023] Chenxing Jiang, Hanwen Zhang, Peize Liu, Zehuan Yu, Hui Cheng, Boyu Zhou, and Shaojie Shen.  H2-mapping: Real-time dense mapping using hierarchical hybrid representation.  _IEEE Robotics and Automation Letters_ , 8(10):6787–6794, 2023.
  * Jin et al. [2023] Liren Jin, Xieyuanli Chen, Julius Rückin, and Marija Popović.  Neu-nbv: Next best view planning using uncertainty estimation in image-based neural rendering.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2023.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics_ , 42(4), 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kopanas and Drettakis [2023] Georgios Kopanas and George Drettakis.  Improving NeRF Quality by Progressive Camera Placement for Free-Viewpoint Navigation.  In _Vision, Modeling, and Visualization_. The Eurographics Association, 2023.
  * Lee et al. [2022] Soomin Lee, Chen Le, Wang Jiahao, Alexender Liniger, Suryansh Kumar, and Fisher Yu.  Uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields.  _IEEE Robotics and Automation Letters_ , 2022.
  * Li et al. [2023] Ke Li, Tim Rolff, Susanne Schmidt, Reinhard Bacher, Wim Leemans, and Frank Steinicke.  Interacting with neural radiance fields in immersive virtual reality.  In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ , New York, NY, USA, 2023. Association for Computing Machinery.
  * Liu et al. [2022] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang.  Neural rays for occlusion-aware image-based rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7824–7833, 2022.
  * Liu et al. [2023] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang.  Robust dynamic radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 41(4):102:1–102:15, 2022.
  * Novkovic et al. [2020] Tonci Novkovic, Remi Pautrat, Fadri Furrer, Michel Breyer, Roland Siegwart, and Juan Nieto.  Object finding in cluttered scenes using interactive perception.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 8338–8344. IEEE, 2020.
  * Ojala et al. [2002] T. Ojala, M. Pietikainen, and T. Maenpaa.  Multiresolution gray-scale and rotation invariant texture classification with local binary patterns.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 24(7):971–987, 2002.
  * Pan et al. [2022] Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang.  Activenerf: Learning where to see with uncertainty estimation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ , pages 230–246. Springer, 2022.
  * Pito [1999] R. Pito.  A solution to the next best view problem for automated surface acquisition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 21(10):1016–1030, 1999.
  * Ran et al. [2023] Yunlong Ran, Jing Zeng, Shibo He, Jiming Chen, Lincheng Li, Yingfeng Chen, Gimhee Lee, and Qi Ye.  Neurar: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations.  _IEEE Robotics and Automation Letters_ , 8(2):1125–1132, 2023.
  * Reddi et al. [2018] Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar.  On the convergence of adam and beyond.  In _International Conference on Learning Representations_ , 2018.
  * Ren et al. [2019] Xinyi Ren, Jianlan Luo, Eugen Solowjow, Juan Aparicio Ojea, Abhishek Gupta, Aviv Tamar, and Pieter Abbeel.  Domain randomization for active pose estimation.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 7228–7234. IEEE, 2019.
  * Rosinol et al. [2022] Antoni Rosinol, John J Leonard, and Luca Carlone.  Nerf-slam: Real-time dense monocular slam with neural radiance fields.  _arXiv preprint arXiv:2210.13641_ , 2022.
  * Shen et al. [2021] Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguer.  Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations.  In _2021 International Conference on 3D Vision (3DV)_ , pages 972–981. IEEE, 2021.
  * Tancik et al. [2023] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa.  Nerfstudio: A modular framework for neural radiance field development.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023.
  * Wang et al. [2020] Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun.  V2vnet: Vehicle-to-vehicle communication for joint perception and prediction.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pages 605–621. Springer, 2020.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Wynn and Turmukhambetov [2023] Jamie Wynn and Daniyar Turmukhambetov.  DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models.  In _CVPR_ , 2023.
  * Xu et al. [2023] Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Aljaž Božič, Dahua Lin, Michael Zollhöfer, and Christian Richardt.  VR-NeRF: High-fidelity virtualized walkable spaces.  In _SIGGRAPH Asia Conference Proceedings_ , 2023.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021.
  * Yu et al. [2023] Jason J. Yu, Fereshteh Forghani, Konstantinos G. Derpanis, and Marcus A. Brubaker.  Long-term photometric consistent novel view synthesis with diffusion models.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * Zeng et al. [2020] Rui Zeng, Wang Zhao, and Yong-Jin Liu.  Pc-nbv: A point cloud based deep network for efficient next best view planning.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7050–7057. IEEE, 2020.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhu et al. [2023] Chengxuan Zhu, Renjie Wan, Yunkai Tang, and Boxin Shi.  Occlusion-free scene recovery via neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 20722–20731, 2023.
  * Zhu et al. [2022] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys.  Nice-slam: Neural implicit scalable encoding for slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.

Supplementary Material

##  6 Supplementary

As supplementary, we provided more details of our proposed approach below. In
Sec. 6.1, we discuss about the dataset generation procedure used to generate
our custom Objaverse [7] based dataset. Then in Sec. 6.2 we outlined the
detailed configurations we used for the respective radiance fields model and
baseline. Thereafter in Sec. 6.3, a further drill-down of the results
previously shown in Tab. 3 of Sec. 4.2 is provided. Finally, we provided a
video visualizing the trajectories and resulting novel view synthesis renders
of our method, SO-NeRF against ActiveNeRF [33] in Sec. 6.4.

###  6.1 Dataset Generation

Figure 4: Visualization of the camera poses visited to sample views for the train, validation and test sets of the cone object. |  PSNR ($\uparrow$) |  LPIPS ($\downarrow$) |  SSIM ($\uparrow$)
---|---|---|---
Class | Rand | E | Op. CQDT | Rand | E | Op. CQDT | Rand | E | Op. CQDT
bookcase | 28.94 | 30.37 | 30.42 | 0.089 | 0.078 | 0.074 | 0.946 | 0.956 | 0.957
chair | 29.33 | 29.59 | 30.00 | 0.084 | 0.069 | 0.067 | 0.949 | 0.956 | 0.957
cone | 27.64 | 28.37 | 28.59 | 0.100 | 0.088 | 0.083 | 0.938 | 0.945 | 0.947
control | 29.33 | 30.41 | 30.48 | 0.069 | 0.055 | 0.055 | 0.957 | 0.965 | 0.965
earphone | 29.16 | 29.74 | 30.05 | 0.074 | 0.060 | 0.058 | 0.954 | 0.961 | 0.961
fan | 27.30 | 27.01 | 26.98 | 0.091 | 0.092 | 0.093 | 0.939 | 0.938 | 0.938
fireplug | 30.21 | 30.37 | 31.30 | 0.064 | 0.065 | 0.054 | 0.959 | 0.959 | 0.964
globe | 25.54 | 26.05 | 26.14 | 0.110 | 0.094 | 0.089 | 0.915 | 0.925 | 0.928
gravestone | 31.43 | 31.52 | 31.59 | 0.054 | 0.050 | 0.051 | 0.967 | 0.968 | 0.969
guitar | 32.22 | 32.43 | 32.33 | 0.053 | 0.045 | 0.045 | 0.976 | 0.979 | 0.979
lion | 30.53 | 30.70 | 30.57 | 0.063 | 0.053 | 0.052 | 0.962 | 0.965 | 0.965
shoe | 26.23 | 26.97 | 26.73 | 0.099 | 0.085 | 0.085 | 0.931 | 0.937 | 0.937
snowman | 32.01 | 32.49 | 32.62 | 0.063 | 0.057 | 0.056 | 0.966 | 0.970 | 0.971
soccer_ball | 25.53 | 24.24 | 25.10 | 0.113 | 0.109 | 0.097 | 0.941 | 0.939 | 0.944
sunglasses | 28.26 | 28.01 | 28.61 | 0.093 | 0.079 | 0.072 | 0.955 | 0.958 | 0.962
teddy_bear | 30.92 | 31.44 | 31.53 | 0.064 | 0.055 | 0.054 | 0.960 | 0.965 | 0.965
toilet | 28.95 | 29.53 | 30.13 | 0.098 | 0.086 | 0.077 | 0.943 | 0.950 | 0.954
vase | 29.75 | 30.92 | 30.97 | 0.073 | 0.063 | 0.064 | 0.952 | 0.959 | 0.959
Table 4: Reconstruction quality of TensoRF trained via trajectory generated
from Optimal CQDT and Ensemble, E against the Random baseline. Results here
are grouped by object class and averaged across the budgets. The best result
is represented with bold.

To begin, we generated our custom dataset using the Objaverse API [7]. Given
the large variety of classes and object instances per class that are available
within the Objaverse dataset, there is an expected amount of noise to the
labels. For this reason, we specifically select only Large Vocabulary Instance
Segmentation [15] (LVIS) objects within Objaverse, which have been manually
verified by humans.

Just like the NeRF Synthetic dataset, our custom data generation procedure
samples views from poses around the object constrained to a fixed radius; this
is equivalent to sampling views from the surface of an imaginary sphere. For
the training and validation set, we performed the sampling randomly around the
object, whereas for the test set, we used a smooth circular trajectory instead
for ease of viewing. An illustration of the aforementioned sampling procedures
can be found in Fig. 4. The procedure is summarized in Algorithm 2. We ran the
algorithm on a machine with a single AMD Radeon MI50 GPU, $4$ CPUs and 16GB of
DRAM.

Overall, we sampled 640 objects across $20$ classes - specifically, bookcase,
chair, cone, control, die, earphone, fan, fireplug, fighter_jet, globe,
gravestone, guitar, lion, shoe, snowman, soccer_ball, sunglasses, teddybear,
toilet, and vase from the list of available categories. For each class, we
downloaded twenty instances of that class and manually verified the labels for
reliability. We pruned or replaced any outlier objects. Visualizations of the
object classes can be found in Fig. 5.

###  6.2 Additional Model Configuration Details

For radiance fields modeling, we leveraged the Nerfstudio [40] implementations
of TensoRF [6] and Vanilla NeRF [29] for training. For TensoRF, we trained
each object-instance for $300$ epochs with all other configurations left as
default. For Vanilla NeRF, we trained each object-instance for 3,000 epochs
instead for better convergence, with all other configurations left as default
as well.

Figure 5: Preview of an object from each class of our custom Objaverse dataset
used to train and evaluate SOARNet. Algorithm 2 ProcessGLB

1:procedure ProcessGLB

2: $\textsc{CleanScene}($) $\triangleright$ Clean the blender scene

3: $obj\leftarrow\textsc{LoadObject}(glb\\_file\\_path)$

4: $\textsc{ScaleObject}(obj)$ $\triangleright$ Normalize scale

5: $imgs,cam\\_parameters\leftarrow\textsc{SampleViews}($)

6: $\textsc{SaveObj}(obj)$ $\triangleright$ Export obj + mtl file

7: $\textsc{SaveSamples}(imgs,cam\\_parameters)$

For Plenoxels [10], we trained it with an epoch-size of 300 per object, with a
learning rate of 3e1, and spherical harmonics learning rate of 1e-2 with all
other configurations as default. In terms of hardware, both models were
trained on a platform with an NVIDIA RTX8000 GPU, 32GB of DRAM, and $2$ CPUs.

For our ActiveNeRF baseline, all the trajectory planning and generation runs
were conducted on a machine with an NVIDIA RTX8000 GPU, 64GB of system DRAM,
and 8 CPUs for training, per object. We initialized each instance with a seed
training set with $5$ poses and let ActiveNeRF determine the required
remaining images based on the quantified uncertainty. Closely following the
experimental configuration of [33], we trained the model for 200,000
iterations each, and let it pick the remaining $B-5$ number of images over the
course of those iterations at a cadence of 5 images per 40,000 steps and
minimum uncertainty kept at 0.01. The remaining settings were all kept at
default.

###  6.3 Additional Result Breakdown

In Sec. 4.2, the averaged results on the Objaverse dataset were reported in
Tab. 3. In this section, we go one step further by breaking down the result
per-class to provide better clarity on the performance of our proposed
approach - Ensemble and Optimal CQDT as previously defined - against the
Random baseline in Tab. 4.

As we can observe from the table, the proposed approaches worked the best in
almost all cases, except for soccer_ball and fan, across all 3 metrics. The
results shown here affirm the findings and conclusion we put forth previously.

###  6.4 Video Rendering

For additional visualization and demonstration of our approach, we have
included a short video together with the supplementary submission to
illustrate the trajectory generated and test-set renderings of SO-NeRF against
ActiveNeRF on the NeRF Synthetic dataset.


#  $p^{\infty}$–Selmer ranks of CM Abelian Varieties

Jamie Bell  University College London, Gower Street, London, WC1E 6BT, UK
<EMAIL_ADDRESS>
###### Abstract.

For an elliptic curve with complex multiplication over a number field, the
$p^{\infty}$–Selmer rank is even for all $p$. Česnavičius proved this using
the fact that $E$ admits a $p$-isogeny whenever $p$ splits in the complex
multiplication field, and invoking known cases of the $p$-parity conjecture.
We give a direct proof, and generalise the result to abelian varieties.

###### 2020 Mathematics Subject Classification:

11G15, 14K22

## Introduction

An elliptic curve over a number field with endomorphisms other than
multiplication by an integer is said to have complex multiplication. These can
make the curves easier to work with. It is well known, and important for this
paper, that they always have even rank. Indeed, suppose we have an elliptic
curve $E$ over $K$ with complex multiplication, and note that
$E(K)\otimes_{\mathbb{Z}}\mathbb{Q}$ is a vector space over $\mathbb{Q}$ with
dimension equal to the rank of $E$. Now the endomorphisms must in fact form a
lattice in an imaginary quadratic field $L$. Then
$E(K)\otimes_{\mathbb{Z}}\mathbb{Q}$ is an $L$–vector space, and so an even-
dimensional $\mathbb{Q}$–vector space.

It is not hard to prove that these curves also have root number 1 ([1], Prop.
6.3), so they satisfy the parity conjecture. We might hope that there is an
analogous result for Selmer groups. This is in fact true, however it is not so
easy to prove. In this paper we will present a new proof of this result, and
generalise it to abelian varieties (Theorem 1). To see why the analogous proof
fails, note that we are now looking at $\mathbb{Q}_{p}$–vector spaces. Suppose
$p=5$ and $L=\mathbb{Q}(i)$. Then we can have an action of $L$ on a one-
dimensional $\mathbb{Q}_{p}$–vector space, because $i\in\mathbb{Q}_{5}$. The
method works when $p$ is inert in $L$, but not when it splits.

Suppose we have an elliptic curve $E$ over a number field $K$, which has
complex multiplication. Looking at the Tate–Shafarevich group , we find its
$p$-primary part is isomorphic to $(\textrm{finite
group})\times(\mathbb{Q}_{p}/\mathbb{Z}_{p})^{\delta_{p}}$ for some integer
$\delta_{p}$. We will define the $p^{\infty}$–Selmer rank to be
$\mathrm{rk}_{p}(E)=\mathrm{rk}(E)+\delta_{p}$.

A generalisation of elliptic curves is abelian varieties. The aim of this
paper is to prove the following:

######  Theorem 1.

Suppose $A/K$ is an abelian variety with complex multiplication by $M$, and
$p$ a prime. Then $\mathrm{rk}_{p}(A)$ is even.

Given that the rank is even, Theorem 1 is equivalent to the statement that the
divisible part of has even $\mathbb{Z}_{p}$-corank. This is in fact expected
to be 0, as is conjectured to be finite.

Another reason to expect Theorem 1 to hold is the $p$-parity conjecture, which
states that for an abelian variety $A$ over a number field $K$, with root
number $w(A/K)$,

(1) $(-1)^{\mathrm{rk}_{p}(A/K)}=w(A/K).$

In the CM case, the root number is 1 ([1], Proposition 6.3). In a different
form $p$-parity was conjectured by Selmer in 1954 ([11]). The conjecture is
known in the case where $A$ is an elliptic curve over a number field admitting
a $p$-isogeny thanks to T. and V. Dokchitser ([4]) and Česnavičius ([1]). By
calculating root numbers, this allowed Česnavičius to conclude that for
elliptic curves with complex multiplication, $\mathrm{rk}_{p}(A)$ is even.
However there is no equivalent $p$-parity result to use for abelian varieties,
so we must use a different method.

Throughout the paper we use ‘complex multiplication’ or ‘CM’ to mean complex
multiplication defined over $K$. With CM defined over $\bar{\mathbb{Q}}$, the
$p$-parity conjecture has been proved for elliptic curves over totally real
$K$, but is open in general. For $p\neq 2$ this is due to Nekovar ([8], 5.10)
and for $p=2$ Green and Maistret ([5], 6.5).

From Theorem 1 we can deduce the following:

######  Corollary 2.

Suppose $A$ and $p$ are as in Theorem 1. If $\Sha[p^{\infty}]$ is infinite,
then it contains $(\mathbb{Q}_{p}/\mathbb{Z}_{p})^{2}$.

### Notation

Throughout, we will assume $A$ and $B$ are abelian varieties over a number
field $K$, with a polarisation $\lambda:A\rightarrow\hat{A}$ defined over $K$.

For an isogeny $f$ and a field $L$, denote by $f_{A(L)}$ the map on $L$-points
induced by $f$, and similarly let $f$ by the map induced on the
Tate–Shafarevich group.

We will denote the dual of an abelian variety $A$ by $\hat{A}$, and of an
isogeny $f:A\rightarrow B$ by $\hat{f}:\hat{B}\rightarrow\hat{A}$.

For a prime $p$, $\Sha[p^{\infty}]$ is the $p$-primary part of . $\delta_{p}$
will denote the multiplicity of $\mathbb{Q}_{p}/\mathbb{Z}_{p}$ in
$\Sha[p^{\infty}]$. Specifically, $\Sha[p^{\infty}]\cong(\textrm{finite
group})\times(\mathbb{Q}_{p}/\mathbb{Z}_{p})^{\delta_{p}}$ for some integer
$\delta_{p}$. d will be the divisible part of and nd the quotient of by d.

Write $Y_{p}(A/K)$ for
$\mathrm{Hom}({}_{d}[p^{\infty}],\mathbb{Q}_{p}/\mathbb{Z}_{p})$, the
Pontryagin dual of ${}_{d}[p^{\infty}]$. Let
$\mathcal{Y}_{p}(A/K)=Y_{p}(A/K)\otimes_{\mathbb{Z}_{p}}\mathbb{Q}_{p}$. Note
this is a $\mathbb{Q}_{p}$-vector space of dimension $\delta_{p}$, and an
$\mathrm{End}_{K}(A)\otimes_{\mathbb{Z}}\mathbb{Q}_{p}$-module.

######  Definition 3 (Rosati involution).

For an abelian variety $A$ with polarisation $\lambda$, the Rosati involution
is the involution on $\mathrm{End}_{K}(A)\otimes_{\mathbb{Z}}\mathbb{Q}$
sending $f$ to $f^{{\dagger}}:=\lambda^{-1}\circ\hat{f}\circ\lambda$. We
extend this by continuity to
$\mathrm{End}_{K}(A)\otimes_{\mathbb{Z}}\mathbb{Q}_{p}$.

######  Definition 4 (Complex multiplication).

We say $A$ has complex multiplication if

  * •

$\mathrm{End}_{K}(A)\otimes\mathbb{Q}\supset M$, where $M$ is a totally
complex field containing a totally real field $L$, $[M:L]=2$ and
$[L:\mathbb{Q}]=\mathrm{dim}(A)$

  * •

The Rosati involution corresponds to complex conjugation on $M$.

Note that this is complex multiplication over $K$, not over $\bar{K}$ (which
is sometimes called potential complex multiplication).

### Acknowledgements

I would like to thank my supervisor Vladimir Dokchitser for suggesting this
problem, and for his advice and guidance. I would also like to thank the
reviewers for their comments, and their help in strengthening the result.

This work was supported by the Engineering and Physical Sciences Research
Council [EP/L015234/1], the EPSRC Centre for Doctoral Training in Geometry and
Number Theory (The London School of Geometry and Number Theory) at University
College London.

##  1\. Self-isogenies

Suppose $A$ and $B$ are abelian varieties over a number field $K$, and
$f:A\rightarrow B$ an isogeny between them.

Recall the following theorem.

######  Theorem 5 ([6], Proof of I.7.3, I.7.3.1).

There is some finite set $S$ of places of $K$ such that

(2) $\prod_{v\in
S}\frac{\\#\mathrm{ker}(f_{A(K_{v})})}{\\#\mathrm{coker}(f_{A(K_{v})})}=\frac{\\#\mathrm{ker}(f_{A(K)})}{\\#\mathrm{coker}(f_{A(K)})}\cdot\frac{\\#\mathrm{coker}(\hat{f}_{\hat{B}(K)})}{\\#\mathrm{ker}(\hat{f}_{\hat{B}(K)})}\cdot\frac{\\#\mathrm{ker}(\hat{f})}{\\#\mathrm{ker}(f)}.$

######  Corollary 6.

Suppose that $A=B$, i.e. $f$ is a self-isogeny. Then

(3) $\\#\mathrm{ker}(\hat{f})=\\#\mathrm{ker}(f).$

###### Proof.

In the formula in theorem 5, the left hand side is equal to the ratio of the
volume forms of $A$ and $B$ which appear in the BSD coefficient (see [6]
Section I.7 for a full definition; in the notation of this chapter the volume
term is $\frac{\prod_{\nu\in S}\mu_{\nu}(A,\omega)}{|\mu|^{d}}$). These depend
only on $A$ and not $f$ so when $A=B$, this is 1.

Similarly, the next two terms equal the ratio of the regulators of $A$ and $B$
and the orders of their torsion subgroups. Specifically,

$\frac{\\#\mathrm{ker}(f_{A(K)})}{\\#\mathrm{coker}(f_{A(K)})}\cdot\frac{\\#\mathrm{coker}(\hat{f}_{\hat{B}(K)})}{\\#\mathrm{ker}(\hat{f}_{\hat{B}(K)})}=\frac{\mathrm{Reg}(A/K)\\#B(K)_{tors}\\#\hat{B}(K)_{tors}}{\mathrm{Reg}(B/K)\\#A(K)_{tors}\\#\hat{A}(K)_{tors}},$

therefore when $A=B$ this is also 1. ∎

The following variant of this result will be useful.

######  Lemma 7.

Suppose $f$ is as above, and $p$ any prime. Then

(4)
$\\#\mathrm{ker}(\hat{f}_{\Sha[p^{\infty}]})=\\#\mathrm{ker}(f_{\Sha[p^{\infty}]}),$

###### Proof.

For any prime $p$, the $p$-adic valuations of the kernels in Lemma 6 must be
equal. As both kernels decompose as a product over primes $l$ of their
$l$-primary subgroups, the $p$-part of each side comes from
$\Sha[p^{\infty}]$, so we can replace by $\Sha[p^{\infty}]$ and still have
equality. ∎

######  Lemma 8.

Suppose $f$ is as before. Then we can split the kernels into divisible and
non-divisible parts. Specifically,

(5)
$\\#\mathrm{ker}(\hat{f}_{{}_{d}})\\#\mathrm{ker}(\hat{f}_{{}_{nd}})=\\#\mathrm{ker}(f_{{}_{d}})\\#\mathrm{ker}(f_{{}_{nd}}).$

###### Proof.

We will show
$\\#\mathrm{ker}(f)=\\#\mathrm{ker}(f_{{}_{d}})\\#\mathrm{ker}(f_{{}_{nd}})$
and similarly for $\hat{f}$. This holds by an application of the snake lemma
to the exact sequence

(6) $0\rightarrow{}_{d}\rightarrow\Sha\rightarrow{}_{nd}\rightarrow 0$

with the isogeny $f$, which is valid because $f$ maps d to d. We can also see
that $\\#\mathrm{coker}(f_{{}_{d}})=1$, because $f$ has a conjugate isogeny
$g:A\rightarrow A$. This has the property that $f\circ g=[\mathrm{deg}(f)]$,
and multiplication by an integer is surjective on d. The result follows. ∎

######  Lemma 9.

Let $A/K$ be an abelian variety, $p$ a prime, and $f:A\rightarrow A$ an
isogeny defined over $K$. Then

(7)
$\\#\mathrm{ker}(\hat{f}_{{}_{d}[p^{\infty}]})=\\#\mathrm{ker}(f_{{}_{d}[p^{\infty}]}).$

###### Proof.

By the functoriality and non-degeneracy of the Cassels-Tate pairing on nd,

(8)
$\\#\mathrm{ker}(\hat{f}_{{}_{nd}[p^{\infty}]})=\\#\mathrm{coker}(f_{{}_{nd}[p^{\infty}]})$

([6], proof of I.7.3). Now $\\#\mathrm{coker}(f_{{}_{nd}[p^{\infty}]})$ and
$\\#\mathrm{ker}(f_{{}_{nd}[p^{\infty}]})$ are equal, because
${}_{nd}[p^{\infty}]$ is a finite group. So the non-divisible parts of the
equation in Lemma 8 cancel out, and we have equality of the divisible parts. ∎

##  2\. Complex Multiplication

######  Lemma 10.

Suppose $A/K$ is a polarised abelian variety, $p$ a prime, and $\phi$ an
invertible element of $\mathrm{End}_{K}(A)\otimes_{\mathbb{Z}}\mathbb{Q}_{p}$.
Then

$\mathrm{ord}_{p}\mathrm{det}(\phi_{\mathcal{Y}_{p}})=\mathrm{ord}_{p}\mathrm{det}(\phi^{{\dagger}}_{\mathcal{Y}_{p}}).$

###### Proof.

We prove this for $\phi$ an isogeny of $A$ defined over $K$, and the full
result follows. By properties of Pontryagin duality,

$\\#\mathrm{ker}(\phi_{{}_{d}[p^{\infty}]})=\\#\mathrm{coker}(\phi_{Y_{p}}).$

Now $\phi_{Y_{p}}$ can be represented over $\mathbb{Z}_{p}$ by a matrix $P$ in
Smith normal form, with all diagonal entries non-zero. Then

$\mathrm{ord}_{p}\mathrm{det}(\phi_{\mathcal{Y}_{p}})=\mathrm{ord}_{p}\mathrm{det}(P)=\mathrm{ord}_{p}\\#\mathrm{coker}(P)=\mathrm{ord}_{p}\\#\mathrm{coker}(\phi_{Y_{p}}).$

It therefore follows from Lemma 9 that

$\mathrm{ord}_{p}\mathrm{det}(\phi_{\mathcal{Y}_{p}})=\mathrm{ord}_{p}\mathrm{det}(\hat{\phi}_{\mathcal{Y}_{p}}).$

Now as $\hat{\phi}=\lambda\circ\phi^{{\dagger}}\circ\lambda^{-1}$,
$\phi^{{\dagger}}_{\mathcal{Y}_{p}}$ will have the same determinant, and the
result follows. ∎

Suppose from now on that $A$ has complex multiplication by a field $M$, with
totally real subfield $L$. Now $\mathcal{Y}_{p}(A/K)$ is an
$M\otimes_{\mathbb{Q}}\mathbb{Q}_{p}$-module. This is isomorphic to
$\prod_{\mathfrak{p}|p}M_{\mathfrak{p}}$, where the product is over primes
$\mathfrak{p}$ of $M$ lying above $p$, and $M_{\mathfrak{p}}$ is the
completion of $M$ at $\mathfrak{p}$. We can therefore decompose
$\mathcal{Y}_{p}(A/K)$ into a sum of $\mathbb{Q}_{p}$-vector spaces

$\mathcal{Y}_{p}(A/K)=\bigoplus_{\mathfrak{p}|p}V_{\mathfrak{p}},$

where each $V_{\mathfrak{p}}$ is an $M_{\mathfrak{p}}$-vector space.

######  Lemma 11.

For each prime $\mathfrak{p}|p$, we have

$\mathrm{dim}_{\mathbb{Q}_{p}}V_{\mathfrak{p}}=\mathrm{dim}_{\mathbb{Q}_{p}}V_{\bar{\mathfrak{p}}}.$

###### Proof.

If $\mathfrak{p}=\bar{\mathfrak{p}}$, we are done, so suppose they are not
equal. Then define $\alpha$ to be the element of
$M\otimes_{\mathbb{Q}}\mathbb{Q}_{p}=\prod_{\mathfrak{p}|p}M_{\mathfrak{p}}$
which corresponds to $p$ in $M_{\mathfrak{p}}$ and 1 in all the other factors.
Now we can view $\alpha$ as an element of
$\mathrm{End}_{K}(A)\otimes_{\mathbb{Z}}\mathbb{Q}_{p}$. Then

$\mathrm{ord}_{p}\mathrm{det}(\alpha_{\mathcal{Y}_{p}(A/K)})=\mathrm{ord}_{p}\mathrm{det}(p|V_{\mathfrak{p}})=\mathrm{dim}_{\mathbb{Q}_{p}}V_{\mathfrak{p}}.$

Now by the definition of complex multiplication, $\alpha^{{\dagger}}$ acts as
$\bar{\alpha}$. It therefore acts as the identity on $V_{\mathfrak{q}}$ for
$\mathfrak{q}\neq\bar{\mathfrak{p}}$, and as multiplication by $p$ on
$V_{\bar{\mathfrak{p}}}$. So by the same argument we have

$\mathrm{ord}_{p}\mathrm{det}(\alpha^{{\dagger}}_{\mathcal{Y}_{p}(A/K)})=\mathrm{dim}_{\mathbb{Q}_{p}}V_{\bar{\mathfrak{p}}},$

and by Lemma 10 the result follows. ∎

###### Proof of Theorem 1.

It suffices to show that
$\mathrm{dim}_{\mathbb{Q}_{p}}\mathcal{Y}_{p}(A/K)=\sum_{\mathfrak{p}|p}\mathrm{dim}_{\mathbb{Q}_{p}}V_{\mathfrak{p}}$
is even. Let $L$ be the fixed field of complex conjugation on $M$. If
$\mathfrak{p}$ is inert or ramified in $M/L$, then
$[M_{\mathfrak{p}}:\mathbb{Q}_{p}]$ is even. Therefore

$\mathrm{dim}_{\mathbb{Q}_{p}}V_{\mathfrak{p}}=[M_{\mathfrak{p}}:\mathbb{Q}_{p}]\mathrm{dim}_{M_{\mathfrak{p}}}V_{\mathfrak{p}}$

is also even.

For the primes $\mathfrak{p}$ which split in $M/L$, we have
$\mathfrak{p}\neq\bar{\mathfrak{p}}$, and

$\mathrm{dim}_{\mathbb{Q}_{p}}V_{\mathfrak{p}}=\mathrm{dim}_{\mathbb{Q}_{p}}V_{\bar{\mathfrak{p}}}.$

Thus $\sum_{\mathfrak{p}|p}\mathrm{dim}_{\mathbb{Q}_{p}}V_{\mathfrak{p}}$ is
even, and so is $\mathrm{rk}_{p}(A/K)$. ∎

######  Remark 12.

The complex multiplication assumption can be weakened. Suppose $A$ is an
abelian variety with $\mathrm{End}_{K}(A)\otimes_{\mathbb{Z}}\mathbb{Q}\supset
M$, for some field $M$, and suppose the Rosati involution induces a non-
trivial automorphism on $M$. Then we can still show that $\mathrm{rk}_{p}(A)$
is even. Here denote this automorphism by $\phi\mapsto\bar{\phi}$, and the
fixed field of it plays the role of $L$. Then the proof proceeds in the same
way.

######  Remark 13.

A similar argument can also be applied directly to $p^{\infty}$-Selmer groups
instead of d. Let
$X_{p}(A/K)=\mathrm{Hom}(\mathrm{Sel}_{p^{\infty}}(A/K),\mathbb{Q}_{p}/\mathbb{Z}_{p})$
and $\mathcal{X}_{p}(A/K)=X_{p}(A/K)\otimes_{\mathbb{Z}_{p}}\mathbb{Q}_{p}$.
Note that the $\mathbb{Q}_{p}$-rank of $\mathcal{X}_{p}(A/K)$ is
$\mathrm{rk}_{p}(A/K)$. Then replace Theorem 5 with Theorem 7 from [3]. This
tells us that for any self-isogeny $\phi$, $Q(\phi)=Q(\hat{\phi})$, where, for
an isogeny $\psi:A\rightarrow B$,

$Q(\psi):=|\mathrm{coker}(\psi:A(K)/A(K)_{\mathrm{tors}}\rightarrow
B(K)/B(K)_{\mathrm{tors}})|\times|\mathrm{ker}(\psi:\Sha(A)_{\mathrm{div}}\rightarrow\Sha(B)_{\mathrm{div}})|.$

Section 2 of [2] tells us that

$\mathrm{ord}_{p}Q(\phi)=\mathrm{ord}_{p}|\mathrm{coker}(\phi:X_{p}(A/K)\rightarrow
X_{p}(A/K))|.$

By the same arguments as in the proof of Lemma 10, we can show that for any
invertible $\phi\in\mathrm{End}_{K}(A)\otimes_{\mathbb{Z}}\mathbb{Q}_{p}$,

$\mathrm{ord}_{p}\mathrm{det}(\phi_{\mathcal{X}_{p}})=\mathrm{ord}_{p}|\mathrm{coker}(\phi_{X_{p}})|=\mathrm{ord}_{p}|\mathrm{coker}(\hat{\phi}_{X_{p}})|=\mathrm{ord}_{p}\mathrm{det}(\hat{\phi}_{\mathcal{X}_{p}})=\mathrm{ord}_{p}\mathrm{det}(\phi^{{\dagger}}_{\mathcal{X}_{p}}).$

Then by an argument similar to Lemma 11 and the surrounding discussion,
$\mathrm{rk}_{p}(A/K)=\mathrm{dim}_{\mathbb{Q}_{p}}(\mathcal{X}_{p}(A/K))$ is
even.

## References

  * [1] K. Česnavičius.  The p-parity conjecture for elliptic curves with a p-isogeny.  Journal für die reine und angewandte Mathematik, vol. 2016, no. 719, pp.45–73 (2016).
  * [2] T. Dokchitser and V. Dokchitser  Self-duality of Selmer groups Mathematical Proceedings of the Cambridge Philosophical Society, 146 (2009), 257–267
  * [3] T. Dokchitser and V. Dokchitser  On the Birch-Swinnerton-Dyer quotients modulo squares Annals of Mathematics, 172 (2010), 567–596
  * [4] T. Dokchitser and V. Dokchitser  Root numbers and parity of ranks of elliptic curves Journal für die reine und angewandte Mathematik, vol. 2011, no. 658, pp.39-64 (2011).
  * [5] H. Green and C. Maistret  The 2-parity conjecture for elliptic curves with isomorphic 2-torsion Proceedings of the Royal Society A, 478, 2022, http://doi.org/10.1098/rspa.2022.0112
  * [6] J. S. Milne, Arithmetic duality theorems, Perspectives in Mathematics, No. 1, Academic Press, 1986.
  * [7] J. Nekovář  Selmer complexes Astérisque 310 (2006).
  * [8] J. Nekovář  Compatibility of arithmetic and algebraic local constants (the case $l\neq p$) Compos. Math. 151 (2015), no. 9, 1626-1646
  * [9] K. Rubin.  Elliptic curves with complex multiplication and the conjecture of Birch and Swinnerton-Dyer.  Arithmetic theory of elliptic curves (Cetraro, 1997), Lecture Notes in Math., vol. 1716, Springer, Berlin, 1999, pp. 167–234, DOI 10.1007/BFb0093455
  * [10] G. Shimura, Abelian Varieties with Complex Multiplication and Modular Functions, Princeton University Press, 1998
  * [11] E. Selmer, A conjecture concerning rational points on cubic curves, Mathematica Scandinavica, 2 (1) (1954), 49–54