#  GovernR: Provenance and Confidentiality Guarantees In Research Data
Repositories

Anwitaman Datta  Nanyang Technological University, Singapore  Chua Chiah Soon
ByteDance, Singapore  Wangfan Gu  Universität des Saarlandes, Germany

###### Abstract

We propose cryptographic protocols to incorporate time provenance guarantees
while meeting confidentiality and controlled sharing needs for research data.
We demonstrate the efficacy of these mechanisms by developing and benchmarking
a practical tool, GovernR, which furthermore takes into usability issues and
is compatible with a popular open-sourced research data storage platform,
Dataverse. In doing so, we identify and provide a solution addressing an
important gap (though applicable to only niche use cases) in practical
research data management.
Keywords: Confidentiality, Data Governance, Data Repository, Integrity,
Provenance, Time-stamping

##  1 Introduction

Openness in research data has direct effect on repeatability and extensibility
of research, with implications on integrity and impact of work. This has led
to well-accepted principles such as FAIR (findability, accessibility,
interoperability, and reusability) data [Wilkinson et al.(2016)]. Many
solutions - third party data hosting services, as well as software to create
institutional repositories - support open hosting of research data.

Notwithstanding the imperatives and advantages of making research data openly
available, there are scenarios where keeping research data confidential is
necessitated. This could be due to the nature of the data itself, e.g., data
carrying personally identifiable information or third party data restricted by
non-disclosure agreements, or driven by other factors such as information to
be withheld pending patent application, trade-secret, etc. A conflict between
confidentiality versus verifiability and provenance (and by implication,
research integrity) results in these circumstances. Foremost, we argue that
while navigating those conflicting needs is non-trivial, it is nevertheless
possible to achieve using standard cryptographic techniques. Even so, it is
impractical for individuals to do so from scratch. Thus, there is a need to
create tools that are easy to use and accessible to users from a wide spectrum
of disciplines unversed in cryptography. Existing tools and systems have
functionality gaps to achieve this, which we address with GovernR.

###  1.1 Premise and Purpose

In the context of information security, the CIA triad encompasses
confidentiality, integrity and availability, which are considered three
cardinal security objectives. The focus of this work spans the former two, and
considers all forms of digital data across arbitrary varieties and formats,
including text, tabular data, media files, documents, code, etc. For the rest
of the paper, we will use the terms data and file(s) interchangeably.

The security objective to restrict disclosure of information only to
authorized entities is termed confidentiality. Specifically, we consider that
it is sometimes desirable to restrict the access to certain data to even
entities within the same organization. Many factors motivate this design
choice, for example: intra-organizational competition, resilience against
insider attacks as well as outsiders who might compromise accounts or IT
assets within an organization, regulatory compliance requirements, e.g., need
to know provisions. In particular, we consider that it may be necessary to
keep the data confidential even from the IT infrastructure administrators
providing and managing an institutional research data repository
infrastructure or service. As such, we distinguish the logical entity, which
we refer to as data owner for a given dataset (which could be one individual,
or a small group of individuals), from the organization which operates the
data repository. Specifically, we are interested in institutional research
data repositories, in which case, typically, the data owner is affiliated to
the organization.

We note that the data owner might at his/her own discretion choose to share
and disclose the dataset to other entities, either within or outside the
organization. As such, preventing entities who already have access to the
dataset from further disseminating it is outside the scope of our work.
Likewise, we assume that if and when the dataset content is audited, the
auditor would have a legitimate need to know the content, and at that time
point, it would (have to) be disclosed to the auditor (but not otherwise, or
earlier). In contrast, to meet provenance requirements, it should be possible
to verify only the time-stamp (of the encrypted data), without divulging the
content.

In terms of data integrity, two sub-objectives arise - (i) verifiability of
the content to be what it has been claimed to be by the data-owner, and (ii)
provenance of the time by when the data already existed (as in, when the data-
owner claims to have populated the data repository with said data instance).
This is essential for audit of the research data, particularly for research
integrity related investigations. Being able to demonstrate the time when the
data already existed is essential in many circumstances, for example, in
establishing priority. In certain contexts, timestamp created within and using
the IT systems of an organization which owns and hosts the data itself may not
be adequate (since the third-party verifier might not trust this information),
particularly if it has to prove the priority to a third party, for instance,
in an intellectual property related dispute.

For data, for which confidentiality is not of concern and can be made
available openly, the integrity objectives can be achieved relatively
trivially (since the data can be scrutinized by anybody from the moment it is
made available). In contrast, for data which has been hosted confidentially,
demonstrating its provenance, particularly if the verifier does not trust the
system timestamp of the hosting service, requires a more nuanced approach.

This is furthermore complicated since the data owner may eventually be unable
or unwilling to facilitate the verification, or may not even be present
anymore, since verification needs may possibly arise many years after the
dataset was created.

Subject to these premises, our objective is to create a mechanism where a
dataset can be hosted on a data repository in an encrypted manner so that the
content remains confidential, yet the data owner can share the content at
his/her discretion to specific others at any time, while integrity of the
dataset in terms of the verifability of the content and time provenance can be
determined whenever an audit is triggered without requiring the auditor to
have access to the content beforehand, or having to trust the hosting service
itself for the timestamp. Essentially, for a future time-point after creation
of the dataset, the data owner needs a mechanism to be able to prove to an
auditor that the encrypted data is indeed the same as the claimed plaintext
data, and it has indeed been present since the time it is claimed to have been
populated in the repository.

###  1.2 Contributions and Paper Organization

This work has two principal contributions. Foremost, we demonstrate how to
leverage existing cryptographic primitives to build a set of protocols that
achieve confidentiality and provenance in terms of content integrity and time-
stamping, addressing a niche nevertheless vital need in research data
governance.

Complementing our work on protocol design, we translate these conceptual ideas
into a practical tool, GovernR, which integrates with a popular open-source
research data repository platform, Dataverse [King(2007)]. Our implementation
accounts for several deployment modes, including one, where the data owner has
exclusive control of who can access the data, but also another, where if the
data owner is not able or available to facilitate data access, the
organization with assistance from a semi-trusted third party (ombudsman) can
also access the data. In this optional mode of operation, the ombudsman is
considered semi-trusted in the sense that it cooperates with the organization
only if there are legitimate reasons to decrypt the data (e.g., for the
purpose of an audit), however neither the ombudsman nor the organization or
system administrators are able to unilaterally access the data.

Next, in Section 2, we discuss related research data repository systems and
also background cryptographic concepts which are leveraged in this work. We
elaborate the GovernR system architecture and associated protocol design in
Section 3, accompanied with discussions on design rationale. In Section 4 we
delve into the details of GovernR implementation and optimizations,
accompanied with discussion of our experiments, which help us validate the
functionalities and benchmark the performance of the current implementation.
We draw our conclusions in Section 5.

GovernR source-code, as well as codes for automating the experiments used for
carrying out the benchmarks are made available at
https://github.com/chiahsoon/DataGovernR.

##  2 Related Works and Background

This literature survey covers two main aspects related to our work: (specific)
related research data repository software, and (general) background
cryptographic techniques used to achieve data privacy and authenticity.

###  2.1 Research Data Repositories

Archival of research data is often necessary to support further study or allow
future auditing of the veracity of the work claimed. To ensure the long term
accessibility and integrity of this archived data, a common approach is for
individuals and organizations to use data repositories. There are many
different types of repositories - public repositories which allow any party to
upload data (e.g. Zenodo111https://zenodo.org/,
figshare222https://figshare.com/) as well as institutional repositories, which
could be based on a third-party software product (e.g. Dataverse [King(2007)],
Dspace333https://dspace.lyrasis.org/, CKAN444https://ckan.org/) or apply
proprietary solutions. We focus on institutional repositories based on third-
party (and open-source) software instead of external proprietary storage and
data management services, and want to leverage on the accessibility of the
source-code, both to build on, and contribute to; given that such software
continue to gain increasing acceptance at research oriented organizations. As
such, we identified prominent software which have strong existing user-base
and has active support and development from the community and list them in
Table 1 along with their key traits and security-related features.

Name | CKAN | Dataverse | DSpace
---|---|---|---
Managed by |  | Open Knowledge
---
Foundation
Harvard IQSS | MIT
Content Focus | Data in general | Research Data | Research Publications
| Role-based
---
Access Control
Yes | Yes | Yes
Embargo | No | Yes | Yes
Versioning | Yes (plugin) | Yes | Yes
Table 1: Popular open-source systems for deploying institutional research data
repositories.

While these solutions offer several security features, they are inadequate for
the issues we have identified. All three systems support Role-Based Access
Control (RBAC), which grants data owners fine control of which users or roles
are allowed to download their uploaded data. In the case of Dataverse, this
can be extremely granular, ranging from making certain files inaccessible (but
still visible), to exposing the basic metadata of a study while hiding its
dataset’s detailed file catalogue. Dataverse and DSpace also support the use
of embargoes, which designate certain files to only become visible after a
certain date. However, these features fall short because the files themselves
are usually stored in the clear on the file system of the server hosting the
repository, meaning that anyone with administrator access could easily view
them. Another relevant feature in the works for Dataverse is support for
Trusted Remote Storage Agents [Crabtree(2018)], where the actual data files
being catalogued by Dataverse are handled by more secure third party storage
providers, such as Amazon’s S3 service. These existing or planned features do
not address the confidentiality issue completely (particularly, even from
users with administrative privilege), nor meet the need of verifiable time-
stamping, particularly for situations where the institution operating the
repository needs to prove the time-stamp to an extrinsic party.

In terms of proving integrity, one common method is to track the version
history of uploaded files and their metadata, so that any changes can be
detected and past versions can be accessed. However, these systems are unable
to cryptographically prove the version history they present. Dataverse also
implements the generation of Universal Numeric Fingerprints
(UNF)555https://github.com/leeper/UNF, which are essentially second pre-image
resistant, (in principle) format-independent digests of normalised data.
However, these are only used for certain tabular data formats inapplicable for
other forms of data.

###  2.2 Cryptography

There are two main uses of cryptography in our work: ensuring the
confidentiality of uploaded data against even an adversary with system
administrator access, and also being able to prove the content integrity and
time-stamp of the data to third parties who may not trust the system’s
internal time-stamps.

####  2.2.1 Maintaining Confidentiality

Encryption is usually used to ensure that uploaded files cannot be accessed by
unauthorised users and is commonly applied by cloud storage services like
Dropbox. This can be done on the client side, which guarantees that the file
cannot be accessed by other parties, or the server side, which can usually
compute encryption and decryption more quickly (although this also usually
requires intermediate encryption of the data in transit). In particular, we
focus on Authenticated Encryption schemes[Bellare and Namprempre(2000)] due to
their use of an authentication tag during the encryption process, which can be
used to verify that the ciphertext and therefore decrypted plaintext have not
been tampered with. However, authenticated encryption alone cannot fully prove
the time of encryption - it can prove that the file was encrypted after a
certain point in time, such as including some associated data that could not
have been known beforehand, e.g., winning lottery numbers, but this would not
prevent an untrustworthy entity from using that same information to create a
fraudulent ciphertext at any later time. Thus, other cryptographic mechanisms
are needed to properly prove the authenticity of archived research data.

####  2.2.2 Provenance

Ultimately, the main use case of our tool is to facilitate the proof of
authenticity of the aforementioned confidential research data to any third
party (who may not trust either the data repository, its operating
organization, or the data owner) that the data owner indeed uploaded the files
in their current state and it was indeed uploaded at the claimed time of
uploading the data.
The simplest such technique relies on using an authority to verify digital
signatures or hashes. Generally, to preserve the privacy of the original
files, a one-way function is used to compute a hash of them to be used in the
time-stamping process. One method is to publish this hash on a publicly
witnessed and dated medium, such as a newspaper advertisement[Bayer et
al.(1993)]. This can also be done using a Public-Key Infrastructure, where a
trusted timestamping authority backs the authenticity of a hash and its
associated timestamp by using its private key to sign the combination of
them[Haber and Stornetta(1990)]. These solutions are currently widely
available and industry accepted, but we wish to reduce the friction of using
these for users who do not specialize in cryptography, making it universally
accessible, and also reduce the reliance (lock-in) on trusting a single
centralized authority.
There are also many distributed solutions for proving the authenticity of a
file. One way is to spread the trust across multiple independent parties (in
this case, they could be separate research institutions) using techniques such
as threshold signature schemes[Shoup(2000)], such that a timestamp cannot
feasibly be forged unless several of them collude to do so. Furthermore,
blockchains have also been found to be a potential source of this trustless
verification, as a timestamp hash minted onto a blockchain ledger cannot be
modified afterwards. This is implemented by various providers such as
OriginStamp[Hepp et al.(2018), Gipp et al.(2015)], which consolidates hashes
from its users and adds them to the blockchain at regular intervals depending
on the fees paid, and supports a freemium model. There have also been more
specific attempts aimed at archival and sharing of research data[Shrestha and
Vassileva(2018)][Ekblaw et al.(2016)]. While some of the main issues with
blockchain technologies such as the slow transactions and high cost can be
alleviated when applying them to research data archival (e.g. slow transaction
times are not a problem since high time resolution is unnecessary), though
long term sustainability of such services remains unclear. Nevertheless, given
the openness and distributed nature of such public blockchain based approach,
we embrace it in our solution.

Another potentially relevant cryptographic tool is the Proof of Retrievability
[Juels and Kaliski Jr(2007)], a protocol for a client and server to
efficiently verify that a given file is stored on the server and will be
accessible (and therefore has had its integrity maintained) if needed. This
technique could greatly speed up the process of auditing archived research
data, as computing hashes for large files can be very computationally
intensive. Proof of Retrievability has been proposed for use for auditing of
general data stored on the cloud[Sookhak et al.(2014)], although there are
currently no implementations specifically targeting research data.

##  3 System Architecture and Protocols

We have designed a scheme which is able to prove the integrity of archived
research data in terms of timestamping and content while protecting its
confidentiality from any third party as well as the hosting service.

In addition to the end user (which could be a researcher uploading or
retrieving data or someone seeking to verify uploaded data), there are four
main logical components involved. Two of these components together comprise
the GovernR tool, which interact with two third-party components, namely
Dataverse [King(2007)] and OriginStamp [Hepp et al.(2018), Gipp et al.(2015)].
Optionally, two further logical entities may be involved: typically, the
organization hosting the data, and a semi-trusted ombudsman, who might each
store key-shares, where their individual key-shares are inadequate to decipher
the data, but put together, the key to decipher data can be reconstructed even
without the availability or cooperation of the data owner.

Figure 1: GovernR Architecture (and information flow during data upload,
described in Section 3.2.1).

###  3.1 GovernR & Associated Components

The overall GovernR deployment architecture is shown in Figure 1. GovernR
itself has a front-end and a back-end component.

  * •

The Front-end component is an external tool accessible from Dataverse and
provides the majority of the encryption functionalities. Aside from some
communication with the other components, the operations of this tool are run
entirely on the client side browser.

    * –

The Front-end component implements a cryptographically secure hash function
$H:\\{0,1\\}^{*}\rightarrow\\{0,1\\}^{n}$ and an authenticated encryption
scheme $AE:\\{Enc,Dec\\}$. The key generation function that would ordinarily
be used by $AE$ is replaced with a password-based key derivation function
$KGen:\\{0,1\\}^{*}\times\\{0,1\\}^{\text{salt
length}}\rightarrow\\{0,1\\}^{\text{key length}}$. It also creates the
(optional) key-shares for the organization and an ombudsman.

  * •

The Back-end component is an internally hosted server that handles most of the
timestamping functionality used to verify archived data and stores some
information used for file decryption. Note that the Back-end only receives
hashed digests of uploaded research data and never has access to the encrypted
or original files.

Two external components, which provide important services are:

  * •

The Dataverse [King(2007)] installation, which stores and manages archived
research data (encrypted or otherwise) and handles basic user authentication
while providing a user interface.

  * •

The Timestamping service provider, whose timestamps are deemed acceptable for
whichever purposes the timestamping is deemed necessary. The modular design of
our Back-end component means that minimal development is required to support
any such provider, as long as it offers an API to upload a hash and request
proof of its timestamp. In our experiments we used a third party service
(OriginStamp [Hepp et al.(2018), Gipp et al.(2015)]).

Auxiliary components: Shares of the cryptographic key to access a dataset can
be optionally shared with the organization operating the data repository, and
a semi-trusted third party, which we refer as an ombudsman. Either key share
separately cannot be used to reveal any information, yet, put together, they
can be used to retrieve the data even in the absence of the original data
owner’s participation at a future point of time. The ombudsman is trusted not
to share its share with the organization unless there is a legitimate (audit)
trigger. It it not trusted with access to the data itself. Hence, we designate
it as semi-trusted. Legal and logistics considerations for involving such an
ombudsman entity is beyond the scope of our work, but the technological means
for it has been considered.

The end-user storing data with need for confidentiality and provenance needs
only to interact directly with the Dataverse and our Front-end, apart from any
external site used by the Timestamping provider to verify timestamps.

###  3.2 Interaction Protocols

In order for a user to access our tool, s/he needs to go through the following
steps:

  1. 1.

The user navigates to the desired dataset in Dataverse and chooses to Explore
it using our external tool.

     * •

Depending on the status of the dataset, this may require the user to have
certain Dataverse permissions.

  2. 2.

Dataverse redirects the user to our Front-end component and passes on
necessary information such as the ID of the dataset and the user’s API token.
Our Front-end component then uses this information to fetch more details about
the dataset from Dataverse’s API, such as the list of files it contains.

  3. 3.

The Front-end component then uses the Dataverse IDs of these files to fetch
information about them from our Back-end component.

  4. 4.

Finally, the Front-end component displays to the user a list of files within
the chosen dataset, as well as additional information such as whether they had
been encrypted by our tool.

From our Front-end, the user can then carry out a variety of operations
depending on the permissions their Dataverse account has access to.

####  3.2.1 Upload File

One of the most common operations we expect is for the user to upload a file
to be encrypted and timestamped (the interactions and data flow details for
file upload are also shown in Figure 1):

  1. 5.

The user selects one or more files from their device through our Front-end and
chooses to upload them, along with a password $p$.

  2. 6.

For each file uploaded, the Front-end does the following:

    1. (a)

The Front-end uses the password to generate an encryption key $K$, with a
different and random salt $s$ for each file. It then encrypts each file $c$
using each respective key, uploads it to the Dataverse API and receives its
Dataverse file ID as a response.
$s\hskip
2.3pt{\leftarrow\\!\\!\mbox{\tiny${\$}$\normalsize}}\,\\{0,1\\}^{\text{salt
length}}$
$K=KGen(p,s)$ $c=Enc_{K}(m)$.

    2. (b)

The Front-end sends the Dataverse file ID, salt $s$, and hashes of the
original ($H(m)$) and encrypted files ($H(c)$) to the Back-end component.

    3. (c)

The Back-end component computes and uploads a combined hash $h$ to the
Timestamping provider’s API to be notarised, receiving a verification link as
a response.

       * •

Depending on the Timestamping service provider, this could be done immediately
and individually for each file or aggregated into a batch at periodic
intervals (for cost optimization).

       * •

If each file has its own individual timestamp, the hashes of the original and
encrypted file will be concatenated (with unique delimiters for user
convenience) and hashed: $h=H(H(m)||H(c))$

       * •

Similarly, if the hash aggregates multiple files $m_{i}$, we concatenate the
hashes for each file in sequence:
$h=H(H(m_{1})||H(c_{1})||H(m_{2})||H(c_{2})\cdots)$

    4. (d)

The Back-end component then stores the following information about each file
in its database:
i. Dataverse DataFile ID,  ii. Key derivation salt $s$,  iii. Hash of file
$H(m)$,  iv. Hash of encrypted file $H(c)$,  v. Verification link.

####  3.2.2 Download File

A privileged user with access to the dataset (a data owner can leverage on
Dataverse’ native Role Based Access Control (RBAC) mechanism for determining
this) and knowledge of the password for the dataset (data owner(s) may share
necessary information using any out of channel communication) can download an
encrypted file as follows:

  1. 5.

The user selects a file from the Front-end’s list for download and supplies
the password $p$.

  2. 6.

The Front-end downloads the encrypted file $c$ from the Dataverse API.

  3. 7.

Using the Dataverse ID of the file, the Front-end component retrieves all
relevant information about the file from the Back-end component’s database:
i. Key derivation salt $s$,  ii. Hash of file $H(m)$,  iii. Hash of encrypted
file $H(c)$.
The Front-end then uses the password and salt to generate the decryption key
and attempt to decrypt the file: $K=KGen(p,s)$, $m=Dec_{K}(c)$.

     * •

As we use an authenticated encryption scheme, the decryption process involves
verifying the authenticity of the ciphertext. If the wrong password is used to
generate a different key, this process fails and the Front-end returns an
error to the user.

     * •

As a sanity check, the Front-end component also recomputes the hashes $H(m)$
and $H(c)$ and matches them against their values from the Back-end database.

  4. 8.

If there are no errors, the Front-end returns the decrypted file to the user.

Note that, instead of using the password, the owner could also share the
dataset specific key with other users, and a variation of the process
described next can be used instead.

####  3.2.3 Download File (Institutional Access)

As an alternative to requiring the owner’s password (and hence permission) to
decrypt and access an uploaded file, the owner can also enable an optional
secret sharing mechanism. This effectively splits decryption key into two
shares to be held by two distinct parties (e.g. the research institution and
an independent ombudsman), allowing them to decrypt the file without the
password if and only if both agree to do so.

During the upload process:

  1. 6.

After generating the key, the Front-end component splits it into two shares
$q,r$, which are then distributed separately to the relevant parties.
$q\hskip
2.3pt{\leftarrow\\!\\!\mbox{\tiny${\$}$\normalsize}}\,\\{0,1\\}^{\text{key
length}}$
$r=q\oplus K$

During the download process:

  1. 5.

The user selects a file from the Front-end’s list for download and supplies
the shares $q,r$.
(Steps 6 and 7 are identical as the previous description of Download File
process.)

  2. 8.

The Front-end uses the shares to generate the decryption key: $K=q\oplus r$

As with the password-based download process, the use of an authenticated
encryption scheme and verification of the hashes ensure that the file is only
decrypted and returned to the user if the shares are legitimate.

A variation of this approach is for the data owner to share the dataset
specific decryption key $K$ with a third party, for them to download the data
(subject furthermore to the RBAC based access control over the encrypted data
to begin with) and decrypt it.

####  3.2.4 Verify File

Any user can use our tool to verify the timestamp of an uploaded file using
the hash of the encrypted file:

  1. 5.

The user selects a file from the Front-end’s list for verification.

  2. 6.

Using the Dataverse ID of the file, the Front-end component retrieves all
relevant information about the file from the Back-end component’s database:
i. Verification link,  ii. Hash of file $H(m)$,  iii. Hash of encrypted file
$H(c)$.

  3. 7.

The Front-end displays the verification link, which shows a timestamp
verifying when a hash $h$ was uploaded to the Timestamping provider and also
provides the algorithm used to compute $h$ from the constituent hashes (see
Upload File).

  4. 8.

The user returns to Dataverse and downloads the file in its encrypted form,
$c$.

  5. 9.

The user accesses the verification link to view concrete proof of the
timestamped hashes provided by the Timestamping service.

  6. 10.

The user uses the encrypted file to compute the hash $H(c)$, partially
verifying that $h$ is authentic (still having to trust the validity of the
$H(m)$ component of $h$).

     * •

If a user has permission to access the file $m$ (either downloading and
decrypting it through our tool or owning the original file), they can then
compute $H(m)$ and hence verify the entirety of the timestamped hash $h$.

###  3.3 Design Rationale and Security Guarantees

Overall, our system is designed to protect the privacy of uploaded research
data while still allowing those without access privileges to verify that these
exact files had been uploaded at the claimed timestamps.

####  3.3.1 Data Privacy

The privacy of uploaded data is primarily assured by carrying out all
encryption-related operations within the client-side Front-end component, such
that a user can check the source code and monitor network communications to
verify that there is indeed no leakage of data. While the user supplies a
password to the Front-end during the upload and encryption process, the
password is never sent to Dataverse or the Back-end servers - not even in the
hashed form usually seen in password-based authentication. This ensures that
even an adversary with full access to the servers would only have access to
the encrypted ciphertext stored in Dataverse and the (independently and
randomly generated) salt stored in the Back-end database, which minimises
their advantage at recovering the original file assuming that the password and
authenticated encryption scheme used is secure. While the hashes of the
original and encrypted files are publicly available through our Front-end,
this does not leak any information about the original file since a
cryptographically secure hash function is used.

####  3.3.2 Secret Sharing

One important feature we provide is the alternative option of decrypting
uploaded files by combining two split shares of the key. This is often more
practical in the case of third-party audits of archived research data, when
the original researcher who uploaded the data may have left the institution.
It also provides safety against inadvertent loss of the original
password/decryption key by the data owner. In terms of security, the simple
secret sharing scheme used is designed to leak no information about the key to
either parties holding only one share - one share is randomly generated
independent of the key, while the other share is essentially a one-time-pad
encryption which also provides no information about the key on its own.

One possible security concern is that the privacy of all files encrypted using
a key would be permanently compromised if the shares are ever brought together
to reconstruct it. However, since our system generates a new random salt for
each individual file uploaded, while different users (or even different
datasets uploaded by the same user) would ideally be using different
passwords, the likelihood of such a key collision is minimal.

####  3.3.3 Authenticity and Timestamping

The authenticity of uploaded data is guaranteed by timestamping the hash
constructed from the hashes of the original file and the encrypted file, where
using both hashes is the key. Timestamping the hash of the encrypted file
naturally ensures that users without access privileges to the original file
are still able to verify the authenticity of the encrypted file - one of our
main objectives.

Furthermore, including the hash of the original file is necessary due to the
security of the encryption scheme. As the salt used to generate the
encryption/decryption key from the password is never revealed to the user, it
is impossible to prove the relationship between the supposed original file and
the encrypted file. If the encryption scheme used is IND-CPA
(Indistinguishability under Chosen Plaintext Attack) and IND-CCA
(Indistinguishability under Chosen Ciphertext Attack) secure as it should be
(and is the case with our implementation), a user without the key is unable to
gain any information about one file from the other. From an auditor’s point of
view, the ”original file” downloaded could have been fabricated after the
claimed date while the ”encrypted file” found on Dataverse and validated by
the timestamp could just be a pseudorandom string. Thus, the final timestamp
also incorporates the hash of the original file, allowing users with access
privileges to verify the authenticity of the original file.

##  4 Implementation & Experiments

As a proof of concept, we implemented the discussed functionalities in a
Javascript-based app leveraging popular programming libraries. The Front-end
component was developed using the React UI 666https://react.dev/ framework and
the Back-end component was developed using the Express
777https://expressjs.com/ server framework. Both components imported the Forge
package’s Javascript implementations of the underlying cryptographic
primitives: AES-256-GCM for authenticated encryption [McGrew and Viega(2004)],
PBKDF2-HMAC-SHA512 (using 120,000 iterations as recommended by
OWASP888https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)
for key generation from passwords and SHA512 for generation of file hashes.

Outside of our code, we used Dataverse version 5.9 for testing and chose
OriginStamp as our Timestamping provider, which in turn uploads our hashes to
the Ethereum blockchain. An important point to note is that OriginStamp is not
a trusted third-party, but simply a service that provides easy-to-access APIs
for the blockchain. Verification of timestamps can be done via the OriginStamp
dashboard, but also independently by any user via the Block Explorer, or any
blockchain expert by querying a native blockchain node directly.

###  4.1 A Note on the Security of PBKDF2 Primitive

In our work, the same password is reused (for keeping usability simple) by the
data owner for generating the encryption key for different files, but with
file specific salts. As such a potential concern is that if the encryption key
for a file and the salt used to generate it are known, is it possible to
recover the password or derive the key of other files with known salts? We are
not aware of any such current vulnerability of PBKDF2, though, at a future
point of time, vulnerabilities may be discovered and exploited. Having said
that, PBKDF2 is in fact part of RSA Laboratories’ Public-Key Cryptography
Standards (PKCS), and it is also published as Internet Engineering Task
Force’s RFC999https://www.rfc-editor.org/rfc/rfc801; as such, our choice is
robust and in lines with deploying cryptographic tools in practice.

###  4.2 Performance Benchmarks

The efficacy and practicality of the proposed approach needs to be determined
in terms of both the achieved security guarantees (discussed in Section 3.3)
as well as the performance of the various functions of the resulting system.
An expected downside of implementing cryptographical features is that they
tend to take their toll on performance. In order to quantify GovernR’s impact
on the end user experience, we benchmarked file upload and download operations
and broke down the runtimes for each major underlying procedure, which we
report next.

####  4.2.1 Methodology

We took benchmarks of the file upload and file download operations, as these
are likely to be the most common operations carried out for data archival
purposes. To estimate the time complexity of our tool with respect to file
size, we carried out these tests for 3 file sizes of 1 MB, 100 MB and 1000 MB,
as well as both a tabular format (comma-separated values) and non-tabular
format (markdown) to take into account how Dataverse handles these file
formats differently. Each combination of operation type, file size and file
format was tested multiple times, with the run times averaged for statistical
analysis. Puppeteer101010https://github.com/puppeteer/puppeteer was used to
automate the testing.

###  4.3 Implementation Optimisations

From the onset of our implementation efforts, we also used the benchmarking
process to identify and optimise slower sections of the code. This gave us a
better idea of how a professional implementation of this functionality might
perform and affect the user experience.

  * •

File Handling: One issue with research data is that it tends to be stored in
very large files, which can add significant time and memory overhead to every
processing step we take. Instead of attempting to load the entirety of these
files into memory, we thus used stream processing to ensure that the server
does not get overwhelmed.

  * •

Dataverse Batch Processing: A specific problem we encountered was how
Dataverse handles file uploads. While the Dataverse API is usually capable of
handling multiple file uploads in parallel, it makes an exception for tabular
data files and rejects any further upload requests while it ingests the
tabular data. This meant that we would have to keep polling the API until the
next upload request is accepted, which incurs significant network costs.
Alternatively, each batch of files could be combined into a compressed ZIP
archive to be uploaded all at once, as Dataverse is capable of extracting the
files from the uploaded archive and processing them more efficiently. However,
the compression and decompression process would then take up significant time
and memory, which was further exacerbated by the fact that decompression
operations become even slower when attempting to extract encrypted data.

  * •

Improving Concurrency using Web
Workers111111https://www.w3schools.com/html/html5_webworkers.asp: The upload
process of our tool involves many operations which could be parallelised to
improve performance. For example, the unencrypted file needs to be hashed for
later use, which can be done at the same time as the file being encrypted.
However, we implemented the tool in the single-threaded JavaScript, which
provides limited concurrency via the Event Loop but not parallelism. Modern
browsers provide Web Workers that run standalone scripts, and they are
typically used to run scripts in parallel to avoid blocking the main UI
thread. In this case, we can offload the hashing work to these web workers.
However, limited memory is allocated to Web Workers. If large files are passed
to them, they are by default cloned, which may cause them to run out of
memory. Additionally, streams cannot be cloned. Web Workers specifies an
alternative to this, which is to transfer the ownership of ‘Transferable‘
objects to the Web Workers. However, streams are also not a ‘Transferable‘
object.
Hence, we implement a simple chunking mechanism to pass chunks to the Web
Workers, instead of passing everything at once. The disadvantage is that it
makes the Web Workers stateful, which can complicate its usage. To make this
slightly more fault-tolerant, the worker script will clear the state if a
‘start’ signal is sent, discarding any previous incomplete state. For example,
in the scenario that the main UI thread dies before the hashing is complete,
the next time the browser tries to upload, the state is cleared instead of
using the discarded state of the previous attempt, which would produce an
invalid hash.

Besides these performance optimizations, in order to reduce cost of time
stamping, we considered aggregation of hashes of datasets across the data
repository using a Merkle tree locally over a period of time, and only time-
stamp the root of the Merkle Tree. This provides a trade-off between the the
granularity at which verifiable time stamps can be created versus the price to
be paid for registering the timestamp with a third party service. Our
experiments were carried out by keeping the frequency of timestamps within the
free limit of the freemium service provided by OriginStamp.

###  4.4 Experiment Results

Note that all figures of time in this section are reported in seconds, rounded
to two most significant digits after decimal. Note also that due to the
rounding of each reported figures, the totals may not add up to the individual
components.

Averaged across both tabular as well non-tabular data considered together, the
(average, rounded) time taken to upload 1MB, 100MB and 1000MB of data using
GovernR are 3.91s, 21.22s and 214.03s respectively, as opposed to 2.41s, 4.00s
and 10.17s for using Dataverse as is. Similarly, downloads take 2.66s, 13.01s
and 116.52s with GovernR against 0.23s, 1.01s and 3.59s for Dataverse.

Tables 2 and 3 provide more granular breakdown across tabular and non-tabular
data, furthermore decomposed into various subtasks involved in the overall
processes. We note that while the key generation process is reasonably
constant and independent of the file size, all other steps such as
cryptographic operations and the file upload/download process increase
linearly with respect to the file size.

The reasons for the drastic reduction in performance are attributed to the
time taken by four main operations: key generation, encryption, hashing, and
sending to Dataverse’s API. Of these operations, the first three are
computation intensive operations which could be improved by choosing more
efficient algorithms or optimising their implementations. As for uploading to
Dataverse, the API endpoint provided by Dataverse does not support
streaming/chunking uploads, so GovernR is also unable to further optimise the
upload time.

Crucially, the time taken to upload the file to Dataverse increases at a
faster rate than the time taken to download a file, which is likely caused by
the issues with Dataverse file handling mentioned in our discussions on
optimisations.

Data type | Tabular | Non-Tabular
---|---|---
Data Size | 1MB | 100MB | 1000MB | 1MB | 100MB | 1000MB
Key Gen | 2.34 | 2.35 | 2.33 | 2.36 | 2.32 | 2.35
Encrypt | 0.29 | 14.72 | 193.59 | 0.28 | 14.40 | 192.16
Zip | 0.28 | 14.72 | 193.60 | 0.28 | 14.41 | 192.17
Plaintext Hash | 0.29 | 14.77 | 193.93 | 0.28 | 14.45 | 192.50
Ciphertext Hash | 0.29 | 14.76 | 193.92 | 0.28 | 14.44 | 192.50
Send to Dataverse | 1.20 | 4.34 | 18.79 | 1.23 | 4.09 | 18.01
Send to GovernR | 0.01 | 0.02 | 0.01 | 0.01 | 0.03 | 0.02
Others | 0.04 | 0.08 | 0.39 | 0.04 | 0.07 | 0.38
Total | 3.89 | 21.52 | 215.13 | 4.42 | 20.93 | 212.93
Table 2: GovernR data upload benchmarks with breakdown of individual classes of operations in seconds (averaged, rounded to two most significant digits after decimal). Data type | Tabular | Non-Tabular
---|---|---
Data Size | 1MB | 100MB | 1000MB | 1MB | 100MB | 1000MB
Key Gen | 2.40 | 2.41 | 2.40 | 2.41 | 2.47 | 2.43
Download from Dataverse | 0.12 | 0.65 | 5.12 | 0.09 | 0.63 | 5.67
Decrypt | 0.13 | 9.93 | 110.48 | 0.13 | 9.86 | 106.88
User Download | 0.13 | 9.93 | 110.48 | 0.13 | 9.86 | 106.88
Others | 0.02 | 0.02 | 0.03 | 0.02 | 0.02 | 0.02
Total | 2.68 | 13.04 | 118.04 | 2.65 | 12.98 | 115.00
Table 3: GovernR data download benchmarks with breakdown of individual classes
of operations in seconds (averaged, rounded to two most significant digits
after decimal).

By testing against all four different scenarios – uploading plaintext file,
encrypted file, zipped plaintext file and zipped encrypted file without the
optimizations from Section 4.3 (Figure 2) we speculate that this additional
time is likely caused by the difficulty of formatting random encrypted data
during the unpacking process. The files tested were across tabular and non-
tabular, hence showing that this additional time affects both types of files.
We can compare these numbers (corresponding to ‘Zipped Encrypted File’)
against the optimized numbers in the Table 2 (‘Send to Dataverse’ row) to see
the extent of improvements that the current optimizations provided, which
remain limited in nature. Recall that our current optimizations were
deliberately non-intrusive, using Dataverse as is, and thus there is room for
improvement in optimizing the implementation of Dataverse in turn, in how it
ingests encrypted data.

Nevertheless, we should also note that it is a small portion of the overall
costs. As such, while current implementation of GovernR not only explores the
protocol designs but provides an actual practical tool with the necessary
functionality of time provenance while conserving content confidentiality, and
does so furthermore taking into account the ease of usability in terms of the
user interface of the front-end which works seamlessly with Dataverse, the
costs of using encryption techniques, unsurprisingly, stays high, which makes
the tool appropriate for only niche usage. This is nevertheless a welcome
first achievement resulting in a working and easy to use system, even as we
open up the pathway for further research in creating more efficient solutions
achieving these security objectives.

Figure 2: Micro-benchmark: Average time taken to send files to Dataverse
(x-axis: MB, y-axis: seconds) without the optimizations described in Section
4.3, considering various combinations of zipping and encryption.

##  5 Concluding remarks

Proper data handling is paramount in science. Even though many tools for
managing research data exist, best practices themselves are still being
ameliorated. We identify the need for guaranteeing data confidentiality while
also ensuring its provenance in terms of content integrity and time provenance
as an important niche for sensitive data. We demonstrate how these disparate
needs can be achieved in conjunction by leveraging mature tools from
cryptography by designing relevant protocols, and implementing a practical
tool that works in a non-intrusive manner with a popular research data
repository.

We validated our tool for functionality and benchmarked its performance. The
user interface and workflow of the tool are accessible to an average user,
without requiring any need for them to know of cryptography, nor complicating
the overall workflow of typical tasks of data upload, access, etc.

However, unsurprisingly, there is a significant performance penalty because of
the use of computation intensive cryptography, as well as how Dataverse
ingests encrypted data. We did some basic optimizations in the implementation
to mitigate these to certain extent, nevertheless, for wide-scale usage and
adoption of such a tool, further performance optimization would be necessary.
In that context, the current version of GovernR should be viewed as an initial
stepping stone. Given that our software is open-sourced and modular in nature,
it allows the wider community to organically improve individual functions over
time. This would include the use of different and more efficient cryptographic
libraries, design a fundamentally different protocol altogether, as well as
optimize encrypted data ingestion at Dataverse end.

## Source-code

GovernR source-code, as well as codes for automating the experiments used for
carrying out the benchmarks are made available at
https://github.com/chiahsoon/DataGovernR.

## Acknowledgements & Attributions

The research presented in this work was supported by Ministry of Education
(MOE) Singapore’s Tier 1 Research Integrity Grant 2020 Award Number
020583-00001, and Ministry of Education (MOE) Singapore’s Tier 2 Grant Award
Number MOE-T2EP20120-0003.

We thank the organizers and participants of the Dataverse Community Meeting,
and Jim Myers in particular, for their patient hearing of and feedback on our
ideas in the early stages of this work.

Chua Chiah Soon contributed to this work when he was a student at NTU
Singapore. Wangfan Gu contributed to this work when he was working as a
project officer at NTU Singapore.

## References

  * [1]
  * [Bayer et al.(1993)] Dave Bayer, Stuart Haber, and W Scott Stornetta. 1993.  Improving the efficiency and reliability of digital time-stamping.  In _Sequences Ii_. Springer, 329–334.
  * [Bellare and Namprempre(2000)] Mihir Bellare and Chanathip Namprempre. 2000.  Authenticated encryption: Relations among notions and analysis of the generic composition paradigm. In _International Conference on the Theory and Application of Cryptology and Information Security_. Springer, 531–545.
  * [Crabtree(2018)] Jon Crabtree. 2018\.  Making Private Data Discoverable: A Trusted Remote Storage Agent for Dataverse.  https://cyberimpact.us/making-private-data-discoverable-a-trusted-remote-storage-agent-for-dataverse/
  * [Ekblaw et al.(2016)] Ariel Ekblaw, Asaph Azaria, John D Halamka, and Andrew Lippman. 2016.  A Case Study for Blockchain in Healthcare:“MedRec” prototype for electronic health records and medical research data. In _Proceedings of IEEE open & big data conference_, Vol. 13. 13.
  * [Gipp et al.(2015)] Bela Gipp, Norman Meuschke, and André Gernandt. 2015\.  Decentralized trusted timestamping using the crypto currency bitcoin.  _arXiv preprint arXiv:1502.04015_ (2015).
  * [Haber and Stornetta(1990)] Stuart Haber and W Scott Stornetta. 1990.  How to time-stamp a digital document. In _Conference on the Theory and Application of Cryptography_. Springer, 437–455.
  * [Hepp et al.(2018)] Thomas Hepp, Alexander Schoenhals, Christopher Gondek, and Bela Gipp. 2018.  OriginStamp: A blockchain-backed system for decentralized trusted timestamping.  _it-Information Technology_ 60, 5-6 (2018), 273–281.
  * [Juels and Kaliski Jr(2007)] Ari Juels and Burton S Kaliski Jr. 2007.  PORs: Proofs of retrievability for large files. In _Proceedings of the 14th ACM conference on Computer and communications security_. 584–597.
  * [King(2007)] Gary King. 2007\.  An introduction to the dataverse network as an infrastructure for data sharing.  _Sociological Methods & Research_ 36, 2 (2007), 173–199.
  * [McGrew and Viega(2004)] David McGrew and John Viega. 2004.  The Galois/counter mode of operation (GCM).  _submission to NIST Modes of Operation Process_ 20 (2004), 0278–0070.
  * [Shoup(2000)] Victor Shoup. 2000\.  Practical threshold signatures. In _International Conference on the Theory and Applications of Cryptographic Techniques_. Springer, 207–220.
  * [Shrestha and Vassileva(2018)] Ajay Kumar Shrestha and Julita Vassileva. 2018.  Blockchain-based research data sharing framework for incentivizing the data owners. In _International Conference on Blockchain_. Springer, 259–266.
  * [Sookhak et al.(2014)] Mehdi Sookhak, Adnan Akhunzada, Abdullah Gani, Muhammad Khurram Khan, and Nor Badrul Anuar. 2014\.  Towards dynamic remote data auditing in computational clouds.  _The Scientific World Journal_ 2014 (2014).
  * [Wilkinson et al.(2016)] Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, et al. 2016\.  The FAIR Guiding Principles for scientific data management and stewardship.  _Scientific data_ 3, 1 (2016), 1–9.


We present a new method that proposes the next best camera placement for NeRF
capture (left). We introduce two metrics that can be easily computed,
observation frequency and angular uniformity (middle). On the right, we show
that our approach outperforms two baseline camera placement strategies,
Hemisphere which is the typical approach used in most NeRF methods and Random,
as well as recent related work [PLSH22].

# Improving NeRF Quality by Progressive Camera Placement for Free-Viewpoint
Navigation

Georgios Kopanas10009-0002-5829-2192 and George Drettakis10000-0002-9254-4819
1 Inria & Université Côte d’Azur, France

###### Abstract

Neural Radiance Fields, or NeRFs, have drastically improved novel view
synthesis and 3D reconstruction for rendering. NeRFs achieve impressive
results on object-centric reconstructions, but the quality of novel view
synthesis with free-viewpoint navigation in complex environments (rooms,
houses, etc) is often problematic. While algorithmic improvements play an
important role in the resulting quality of novel view synthesis, in this work,
we show that because optimizing a NeRF is inherently a data-driven process,
good quality data play a fundamental role in the final quality of the
reconstruction. As a consequence, it is critical to choose the data samples –
in this case the cameras – in a way that will eventually allow the
optimization to converge to a solution that allows free-viewpoint navigation
with good quality. Our main contribution is an algorithm that efficiently
proposes new camera placements that improve visual quality with minimal
assumptions. Our solution can be used with any NeRF model and outperforms
baselines and similar work.

<ccs2012> <concept> <concept_id>10010147.10010371</concept_id>
<concept_desc>Computing methodologies Computer graphics</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010371.10010372</concept_id> <concept_desc>Computing
methodologies Rendering</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010257.10010282.10011304</concept_id>
<concept_desc>Computing methodologies Active learning settings</concept_desc>
<concept_significance>100</concept_significance> </concept> </ccs2012>

[500]Computing methodologies Computer graphics [500]Computing methodologies
Rendering [100]Computing methodologies Active learning settings

##  1 Introduction

In recent years, Neural Radiance Fields (NeRFs) [TTM∗22] have emerged as a
powerful approach allowing high-quality novel view synthesis, for scenes
captured with photos taken from many different viewpoints. These methods also
provide an alternative to Multi-View-Stereo [SZFP16] solutions that are slow
and fail to produce faithful visual and geometric reconstruction. For both MVS
and NeRF, capturing a scene typically starts with users taking many photos or
video of the scene. Usually, users follow instructions to loop around an
object a few times at different heights and to make sure to also capture top
views [Lum23] which we call _hemispherical_ capture. This works well for
“object-centric scenes”, i.e., scenes that have a main object that the users
want to be able to view freely, while the rest is considered background. There
has been little previous work on how to capture more general scenes such as
rooms, buildings etc. that have no central point of interest, especially when
the goal is to allow _free-viewpoint_ navigation in the environment. Users
typically place the cameras based on their intuition and empirical knowledge
about which camera placements usually work, often leading to the failure of
the reconstruction and consequently, forcing users to recapture the scenes in
a costly and time consuming trial-an-error process.

Intuitively, hemispherical capture works for object-centric scenes because it
samples the space containing the object _uniformly_ both for camera positions
and in the angular domain. This uniform coverage is a dense sampling of a
complete radiance field, since rays from the camera centers through the pixels
in each view frustum cover space uniformly. Such coverage provides multi-view
information that is used in the optimization to disambiguate depth, allowing
accurate reconstruction.

Achieving such uniform ray coverage both in positions and angles is much more
challenging in the context of general scenes, where there is no single central
object. With an infinite number of cameras, it could theoretically be possible
to densely sample the light field, but in practice the number of cameras is
limited and in addition given the geometry of the scene cameras cannot be
placed everywhere (i.e., inside objects). The problem we try to solve is:
given a camera budget and physical limitations of space, how can we
_efficiently_ choose the next camera that will allow the resulting ray
sampling to be as close as possible to uniform in space and angle.

There has been little previous work on this problem; most methods that have
been proposed require either modification on the training of the NeRF model
making them unsuitable for generalization to other NeRF variants other than
the ones that it was specifically designed for, or very expensive calculations
based on the current state of the NeRF model. These properties make the
process slow and cumbersome.

In our solution, we first develop a metric to evaluate uniformity in space and
angle that is fast to evaluate. We then propose an algorithm that uses this
metric to select the next best camera such that the overall distribution will
be closer to uniform in positions and angle. We evaluate the metric and the
algorithm on synthetic data and compare to baselines and previous work,
demonstrating that our solution works well, and we also run our algorithm on a
real dataset as a proof of concept. From a practical perspective, our
algorithm can be used for automated capture using robotic or drone capture; we
leave this as future work, but we discuss a practical future use case in Sec.
3.4.

In summary, our contributions are:

  * •

The definition of an efficient metric for _observation frequency_ and _angular
uniformity_ that can be computed on the fly during NeRF capture, without
requiring additional images.

  * •

An algorithm to quickly estimate reconstruction quality of a scene and that
proposes the next camera placement that maximizes the improvement in quality
of capture based on our metrics.

Our solution can work with any NeRF model without changes to the optimization
loop, and only introduces a small performance overhead to the training. We
performed extensive testing on synthetic scenes, and our method achieves the
best quality against multiple baselines and other algorithms that we tried
given a limited budget of cameras. We also present a first preliminary
evaluation with real data, in which our method also performs well.

##  2 Related Work

In recent years a huge number of publications on Neural Radiance Fields
(NeRFs) have been published; we will start by reviewing only a few papers that
are closely related to our method and design choices. Recent comprehensive
surveys on NeRFs can be found in [TTM∗22] and [XTS∗22]. We then review camera
selection for reconstruction, both traditional and for NeRFs.

###  2.1 Neural Radiance Field Basics

Neural Radiance Fields were introduced by [MST∗21]. They fundamentally changed
how we can reconstruct 3D scenes from 2D images by introducing a continuous
volumetric representation of the scene, encoded in a Multi-Layer Perceptron
(MLP) which we can optimize using Stochastic Gradient Descent (SGD) to fit the
input images, solving the reconstruction with a data-driven optimization.

Follow-up work improved the reconstruction quality by allowing for better
extrapolation when dealing with cameras observing objects from different
distances [BMT∗21] but also generalized the algorithms for unbounded and
realistic scenes [BMV∗22]. Most of the results demonstrated in these papers
focus on object-centric camera placements which has become the “typical NeRF-
style capture”, i.e., a hemisphere of cameras around the object and looking
directly at the center of the object.

Another line of work focuses on performance, both for fast training and fast
rendering. Most solutions achieve good results by encoding the radiance field
in voxel grids with limited spatial extent[MESK22, FKYT∗22, SSC22, CXG∗22] or
point clouds[XXP∗22]. For our experiments we will built on top of Instant-NGP
[MESK22], since it is currently the NeRF method with the best
quality/performance trade-off. Instant-NGP uses a hash function to map a 3D
hierarchical voxel-grid of high dimensional features to a compact 1D
representation. This grid is later queried in an optimized way along the ray
to produce the final color for each pixel. The optimization includes a very
efficient _occupancy grid_ that marks the voxels as occupied or free and
results in an efficient way to skip empty space during ray marching.

Also, there are numerous papers that try to reconstruct NeRFs from a very
limited amount of input views[YYTK21, JTA21, KDSB22]. These models could
potentially be benefited greatly by an optimal selection of cameras.

While all these algorithms significantly improve the state-of-the-art, in the
vast majority of cases they use datasets in which the cameras are placed on a
hemisphere over a region of interest. This allows for good quality
reconstruction only in that specific region (typically an object of interest).
It is not clear how one would place cameras for more complicated environments,
when allowing the user to navigate freely. Camera placement is an important
factor that controls the final quality of the reconstruction and the ability
to _navigate freely_ in the scene without artifacts. In this context we
propose a solution that will automate and standardize the way of capturing
NeRFs, removing the burden of trial and error from the user.

###  2.2 Camera placement for reconstruction

We discuss representative previous work in camera placement for
reconstruction. In the vast majority of cases we assume that the scene is
captured with photographs, and that the cameras of these photographs have been
calibrated. Camera calibration is typically performed using Structure-from-
Motion (SfM), using systems such as COLMAP [SF16].

####  2.2.1 Traditional Reconstruction

Multi-View Stereo (MVS) is an offline process that recovers the 3D geometry
from a set of images and is very computationally expensive. The user first
captures the images, performs camera calibration and then runs MVS. After a
few hours of computation, the user may come to realize that the images are not
good enough for a good 3D reconstruction, requiring the scene to be recaptured
from scratch. This is a tedious process, especially if accessing the capture
site is difficult.

To improve this cumbersome process the field of Next-Best-View (NBV)
estimation [ISDS16, DF09, BKA∗16] predicts the next view that will provide
more information to the reconstruction process given a set of already captured
views. In the field of volumetric reconstruction [ISDS16] focuses on sensors
with depth and creates a set of heuristics to estimate the next best view that
will maximize the information gain of each newly acquired sample. While this
work is inspiring it lies outside the scope of optimizing a model from a set
of data-samples with SGD because they use a depth sensor that directly
observes the geometry information of the scene, while in neural radiance
fields the geometry representation is being optimized to fit the scene.

Other works in camera selection that focus on MVS can be separated in
heurstic-based methods [MRFB16, SMGH18] or data-based [LFR22]. They mostly
focus on estimating or predicting the uncertainty of the MVS reconstruction
process without actually running it. In contrast to MVS, fast NeRF models
[MESK22, FKYT∗22] open the door for new approach in the field of next best
view estimation that allows online reconstruction and camera placement
prediction, especially if camera calibration can be provided online by the
capture device (e.g., augmented reality helmet).

####  2.2.2 Neural Radiance Fields

Automatic camera placement for Radiance Fields is an emerging topic of
research. A popular approach is to modify the NeRF model to be able to predict
it’s own uncertainty [RZH∗23, ZLR∗22, PLSH22] which later is used in various
ways to choose the views which maximize it. The uncertainty is modeled in two
ways, either by converting the MLP that encodes the scene to a Bayesian MLP
[RZH∗23, ZLR∗22, PLSH22] that also predicts it’s own uncertainty or by using
the physical properties of the volumetric representation along a ray based on
the entropy of the density function[ZZX∗22]. All methods that use the NeRF
model to predict uncertainty are computationally intensive since they need
many MLP evaluations for each candidate camera. In addition, it is hard to
train an MLP that predicts it’s own uncertainty. That is why [RZH∗23] uses a
depth sensor to stabilize the training. Some methods [PLSH22] focus on
selecting views when there is a very limited budget of cameras allowed and
[LCW∗22] presents a solution that evaluates the uncertainty based on the
spread of density along a ray. This needs a full rendering step per candidate
camera which means when the space of candidates grows in unconstrained
environments it comes with an increased cost. All the above methods are not
demonstrated on non object-centric scenes, making them unsuitable for our
context which focuses on free-viewpoint navigation in complex scenes.

##  3 Method

The goal of our method is to dynamically suggest new camera positions such
that we create a dataset that will achieve a good quality reconstruction. This
can be used to guide a robotic agent or a human to acquire new images when
capturing a NeRF. NeRFs trained on object-centric datasets achieve excellent
quality when observing the object from a camera that matches the distribution
of the training cameras, but easily break when moving away from them, see Fig.
2. We are interested in constructing a carefully designed placement of cameras
that will allow the final user of the NeRF to navigate freely in the scene,
while avoiding strong visual artifacts.

We want to generalize the simple assumptions of the object-centric capture
style to more complex scenes and viewing scenarios, in particular when we
allow the viewer to navigate freely.

###  3.1 Observation Frequency and Angular Uniformity

The object-centric capture style of NeRF[MST∗21] and MipNeRF[BMT∗21] has two
main properties. First, all cameras observe the object and second, the cameras
are distributed along different directions to cover the angular domain
uniformly. If we constrain the user to view the scene on the hemisphere, this
capture style naturally provides a good reconstruction since it covers the
space all the possible cameras uniformly.

We next provide a formal definition of this observation, and in particular a
measure of _observation frequency_ and a measure of _angular uniformity_ of
these observations.

Given a set of cameras $\mathcal{C}$, a point $p$ in space will be well
reconstructed if it is observed often from the input cameras and if these
cameras are distributed uniformly in the angular domain of directions. We next
formalize this mathematically and generalize it for multiple points $p$.

We define a function that describes how frequently each point is observed. For
a point $p$ we define the frequency $O_{f}(p)$ of observation as follows:

$O_{f}(p)=\frac{\sum_{i=0}^{N}{\mathds{1}_{obs}(\mathcal{C}_{i},p)}}{N}$ (1)

Where $\mathds{1}_{obs}$ is an indicator function that is $1$ if point $p$
lies inside the frustum of camera $C_{i}$ and $N$ is the total number of
cameras. This equation describes a simple relationship between cameras and
points in space: If all the available cameras observe a point, then
$O_{f}(p)=1$, while if no cameras observe it $O_{f}(p)=0$.

Figure 1:  To compute the distance of the empirical distribution in the
angular domain of cameras that observe a point $p$, we assign the cameras to
bins based on their direction from point $p$ into a histogram in polar
coordinates. We then convert this to a PDF, for which we account for the non-
uniform surface area of the spherical coordinate system. Then we use the total
variation distance in Eq.(2) to get the value for node point $p$.

For the directions of the observations, we next define a metric to measure
_angular uniformity_ , since more uniform angular distribution of observed
directions results in better resulting visual quality.

We define $f_{p}$ the distribution of camera directions in the angular domain
that observe a point $p$ and the uniform distribution $u$ in the same angular
domain. To determine the quality of the angular distribution of cameras, we
will compute the total variation distance between the two distributions.

$TV(f_{p},u)=\frac{1}{2}\sum_{\theta,\phi\in\Omega}{|f_{p}(\theta,\phi)-u(\theta,\phi)|}$
(2)

where $u$ is a uniform PDF. We construct the piece-wise constant PDF
$f^{C}_{p}$ in the angular domain by computing the histogram of the directions
of the cameras that observe point $p$. Every bin in the histogram contains the
number of cameras that observe this point from the solid angle that
corresponds to the bin. Similarly to Eq. (1), the angular metric is 0 if our
point p is observed from a uniform distribution, while it approaches 1 as it
moves further from the uniform distribution.

We provide a visual illustration of this process in Fig.1, showing the
histogram of $F_{p}$ and $U$ in bins with polar coordinates. The direction are
represented in polar coordinates, that not area preserving, so we weight the
bins of the histogram based on the surface area of each bin.

###  3.2 Estimating Reconstruction Quality

We define the area where we want to estimate the reconstruction quality, and
for simplicity we use an axis-aligned bounding box to define it. This is the
area which the user wishes to observe; We refer to this area as $\mathcal{B}$.

Ideally, we would like to evaluate the quality of reconstruction of every
point in $\mathcal{B}$. In practice however, we discretize the problem by
constructing a regular grid in $\mathcal{B}$ with resolution of $32^{3}$ and
refer to it as $\mathcal{B}_{32}$; we will evaluate reconstructability on the
_nodes_ of the grid. In discrete space it is easier to measure the total
reconstructability $E$ of $\mathcal{B}$ given by the set of cameras
$\mathcal{C}$ by summing over all the nodes $p$ of the grid:

$E(\mathcal{C},\mathcal{B}_{32})=\sum_{p\in\mathcal{B}_{32}}{(1-TV(f_{p}^{\mathcal{C}},u))+O_{f}(p)^{\gamma}},$
(3)

where $f_{p}^{\mathcal{C}}$ is the empirical PDF in the angular domain for
point $p$ and set of cameras $\mathcal{C}$ and $u$ is the uniform PDF in the
same domain, while $\gamma$ is a non linear scaling factor that modulates how
much more important it is to observe points that are less frequently observed
than points that have already been observed frequently.

Our formulation has several advantages. First, it is easily interpretable by a
user, making it easy to modify and specify regions that have more importance
than others. In the limit case of a single point and if $C$ is constrained on
a hemisphere our method reduces to the typical “object-centric” capture setup
of NeRF and MipNeRF/360. Second, a key advantage is that our camera proposal
does not require additional image acquisition to estimate reconstruction
quality, as for other methods, e.g., based on uncertainty estimation [PLSH22].
Our combined observation frequency and angular uniformity metrics can be seen
as a proxy for uncertainty, while being relatively cheap to compute.

###  3.3 Optimization

Our goal is to find the set of cameras $\mathcal{C}$ that maximize the
quantity in Eq. (3):

$\operatorname*{argmax}_{\mathcal{C}}E(\mathcal{C},\mathcal{B}_{32})$ (4)

To do this we optimize the NeRF model while we choose the new cameras. Cameras
can only be placed in empty space and some NeRF models, including Instant-NGP
[MESK22] that we use, provide an occupancy grid that is used to skip empty
space while rendering. If the implementation does not provide one it is
trivial to compute it on the fly by sampling 3-D space and storing the density
and using a threshold to binarize the value. We use the occupancy grid to
place candidate cameras in free space. For simplicity, we also constrain the
cameras to lie inside $\mathcal{B}$.

Algorithm 1 Summary of the greedy optimization algorithm to minimize Eq. (3

)

$T\leftarrow 100$ $\triangleright$ Total number of cameras we want to sample.

$N\leftarrow 1000$ $\triangleright$ Number of cameras we sample in each
iteration.

$nodes\leftarrow\mathcal{B}_{32}$

$\mathcal{C}\leftarrow[\hskip 2.84544pt]$

while $|\mathcal{C}|<=T$ do

$aabb\leftarrow\mathcal{B}\hskip 5.69046pt\textbf{if}\hskip
5.69046pt|\mathcal{C}|>20\hskip 5.69046pt\textbf{else}\hskip
5.69046pt\mathcal{B}^{f}$

$C_{p}\leftarrow sampleRandomCameras(N,aabb)$

for all $c\in\mathcal{C}_{p}$ do

$E_{c}\leftarrow E(\mathcal{C}\cup c,nodes)$ Eq. (3)

end for

$c_{max}\leftarrow\operatorname*{argmax}_{c}E_{c}$

$C\text{.insert}(AcquireImage(c_{max}))$

if $|\mathcal{C}|>20$ then

$train\\_nerf(iterations=250)$

end if

end while

In the beginning of our run we have no images nor a trained occupancy grid,
and one needs the other to initialize the process. To overcome this, we ask
the user to create a bounding box in the scene that is empty and call it
$\mathcal{B}^{f}$. This allows us to sample cameras safely so we can start the
process. This is somewhat of a chicken-and-egg problem, since we need enough
of an initial reconstruction to have a coordinate system to define this
initial box $\mathcal{B}^{f}$. In a real-world scenario, the user will simply
take 20 photos of the scene – evidently in free space – that will initialize
the reconstruction and the coordinate system, and imply the definition of
$\mathcal{B}^{f}$. For the synthetic examples we used for evaluation, we have
the coordinate system beforehand, and we defined $\mathcal{B}^{f}$ manually.

Now that we know how to sample random cameras safely, we want to maximize the
quantity in Eq. (3). We use a greedy maximization technique: every 250
training iterations we acquire a new camera given a set of already chosen
$\mathcal{C}_{k}$. We do this by sampling a set of N=1000 candidate cameras
that lie inside $\mathcal{B}$, or inside $\mathcal{B}^{f}$ for the first 20
cameras. Each camera gets a random direction, and we filter out all cameras
whose center lies in occupied space or observes occupied space from too close.
Then we compute $E(\mathcal{C}_{k}\cup c_{n},\mathcal{B}_{32})$ for each of
the N cameras, choose the camera with the highest score and add it in
$\mathcal{C}_{k+1}=\mathcal{C}_{k}\cup c_{n}$. We repeat until we acquire as
many cameras as our budget allows. The algorithm is summarized in Alg. 1. Our
current unoptimized implementation requires a few seconds to propose a new
camera; further optimization would allow truly interactive use.

###  3.4 Future Practical Usage Scenario

The method above provides all the elements for online camera selection that
can be used in future work by a robotic or drone-based system. In this
paragraph we describe how such a system could function to motivate the utility
of our results.

As discussed above, a user would first take 10-20 photos of the environment
for an initial camera calibration, providing a reference frame, and allowing
the definition of an initial box $\mathcal{B}^{f}$. In a fully operational
system, we would then run a fast NeRF such as InstantNGP [MESK22] that
reconstructs a first approximation of the scene volumetric representation in a
few seconds. We then would run our algorithm to choose the next camera, and
move the robot or drone to the next position; the new capture is then
incrementally added to the NeRF optimization until the budget of cameras is
reached. Given the latency of moving the capture agent and the fact that we
need a few training iterations between the two captures, a slightly optimized
implementation of our algorithm is perfectly suited to such a scenario, since
it can provide the next best camera faster than the agent can actually move to
the next position. As a consequence, a full system using our algorithm would
allow automatic and high quality capture with a small number of photos, both
reducing capture time and optimizing resulting image quality.

Table 1:  Per Scene Quantitaive evaluation of our method. We provide the PSNR
for each test set separately and the total average of each algorithm for each
scene.

Scene | LivingRoom1 | LivingRoom2 | Office6 | Office9 | Kitchen5
---|---|---|---|---|---
Train|Test | Random | Hsphere | Ours | Avg | Random | Hsphere | Ours | Avg | Random | Hsphere | Ours | Avg | Random | Hsphere | Ours | Avg | Random | Hsphere | Ours | Avg
Hsphere | 17.43 | 31.48 | 14.63 | 21.18 | 19.13 | 30.12 | 13.04 | 20.76 | 20.34 | 30.48 | 15.22 | 22.01 | 15.17 | 36.79 | 13.23 | 21.73 | 18.92 | 33.28 | 16.36 | 22.85
ActiveNerf | 20.57 | 17.91 | 18.59 | 19.02 | 25.51 | 22.76 | 21.28 | 23.18 | 24.71 | 20.78 | 19.65 | 21.71 | 25.75 | 27.52 | 24.40 | 25.89 | 25.00 | 22.83 | 21.42 | 23.08
Random | 25.77 | 23.86 | 22.81 | 24.14 | 26.65 | 24.48 | 22.00 | 24.37 | 28.57 | 25.63 | 22.06 | 25.42 | 26.20 | 28.29 | 22.65 | 25.71 | 25.24 | 24.30 | 22.70 | 24.08
Ours | 27.46 | 27.80 | 26.86 | 27.37 | 28.40 | 27.02 | 26.02 | 27.14 | 31.35 | 27.01 | 27.12 | 28.49 | 28.38 | 30.59 | 28.91 | 29.29 | 27.74 | 27.06 | 25.87 | 26.89

GT | Ours | ActiveNerf[PLSH22] | Random | Hemisphere
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 2:  Images from our sampling test set. We present a visual comparison
to baselines(Random,Hemisphere) and ActiveNerf. The first column shows the
ground truth image. The scenes shown are Livingroom1, Livingroom2, kitchen5,
office6 and office9.

##  4 Results & Comparisons

Evidently, the goal of our method is to provide guidance for a human or
robotic acquisition system while capturing a NeRF. Designing the user
interface for human guidance or interfacing with an automated acquisition
system are complex tasks that we leave for future work. Instead, we provide a
thorough evaluation of synthetic scenes, in which image acquisition is
achieved simply by rendering a new image from the camera proposed by our
system. We also provide a preliminary evaluation of our method on a real
capture, by capturing a large number of views that we can then sample.

This system was implemented by interfacing together Instant-NGP [MESK22] with
Blender’s python API [Com18] and Cycles renderer111https://www.cycles-
renderer.org/. We extended the python interface of Instant-NGP to allow us to
query the occupancy grid efficiently and we also linked Instant-NGP with
Blender’s python environment. We will provide all source code and data, that
will be available here https://gitlab.inria.fr/fungraph/progressive-camera-
placement. The full pipeline that allows the treatment of complex synthetic
scenes interfaced with NeRF systems such as InstantNGP is a powerful tool in
itself, and was very useful for this project. We hope it will also be helpful
to others experimenting with NeRFs in full, realistic scenes.

Figure 3:  “Floorplan” visualisation of the seperate elements that construct
our energy term in Eq. (3). The values have been averaged along the Y axis.

###  4.1 Evaluation on Synthetic Scenes

For the first set of synthetic scene comparisons, we evaluate our method
against two baseline camera placement strategies: Hemisphere where we place
the cameras on a hemisphere around an object of interest222We manually select
a point and a radius for the hemisphere such that it makes for that specific
scene configuration, i.e a table with vases etc in the scene (this is the
standard NeRF[MST∗21] capture style) and Random where we place the cameras at
a random position and a random orientation by also making sure that the camera
placement is not in occupied space (i.e., not inside objects), see also Sec.
3.3.

Figure 4:  In this plot we show the test-set PSNR with relation to the number
of cameras in the training set. We plot two sampling algorithms, random and
ours. We evaluate the PSNR across all 150 images of the test set for the scene
Office6

As discussed in Sec. 2, there are few methods that treat our specific problem;
of all NeRF camera selection methods, only ActiveNerf [PLSH22] provides code,
and we thus include a comparison to this approach. We use the authors’
implementation which is based on an implementation of the original NeRF
[MST∗21] method. To allow a best-effort fair comparison we used their code to
extract the cameras, and then trained the same Instant-NGP[MESK22] model as we
did with all other baselines and our method. To extract the cameras we pre-
render 1000 random cameras which act as the pool from which ActiveNeRF can
choose cameras. The ActiveNeRF implementation is very computationally and
memory intensive so we tuned ActiveNerf to start from 20 random cameras, and
choose 8 cameras every 50k iterations. These 8 cameras are the best cameras
chosen from their algorithm from a random subset of 100 out of the 1000
cameras.

GT | Ours | ActiveNerf[PLSH22] | Random | Hemisphere
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 5:  Images from the random camera sampling test set. We present a
visual comparison to baselines (Random,Hemisphere) and ActiveNerf. The first
column shows the ground truth image. The scenes shown are Livingroom1,
Livingroom2, kitchen5, office6 and office9.

We used 5 synthetic scenes modeled by professional artists to represent
realistic indoor environments333The scenes are available for purchase at
www.evermotion.com and are compatible with the Blender Cycles renderer.. For
each scene we construct a training set corresponding to each one of the
algorithms we want to evaluate and multiple test sets that provide a good
overview of the total quality throughout the scene.

Our test-sets contain a total 150 views that are distinct from the training
views. The test-sets are split in 3 sub-sets: 1) 50 random views using the
Hemisphere capture style 2) 50 views using Random and 3) 50 views using our
sampling process. The purpose of the multiple test sets is to evaluate each
algorithm fairly throughout different camera distributions such that the
quantitative metric evaluate the total quality throughout the scene. This
avoids bias towards one of the aforementioned distributions, and allows a more
comprehensive overall evaluation of our algorithm. We provide all the
renderings for all views and all algorithms in our supplemental material. We
also rendered free-view point paths which we provide in the supplemental
video.

As we observe in Fig. 2 and Fig. 5 the standard hemisphere captures of NeRF
and MipNeRF fail to generalize in test sets coming from other distributions.
Hemisphere views have a specific structure and objects that lie outside of the
hemisphere are observed only from constrained angular directions and this
allows for the NeRF model to overfit to the set of input cameras. While the
random capture significantly outperforms the hemisphere capture in the
generalized setting, we can still see significant artifacts because of the
unstructured nature of the dataset. In theory, an infinite number of random
views should allow for perfect reconstruction, but this is impractical since
it is labor and computationally intensive.

As we can see from the quantitative and qualititave evaluation (Fig. 2,5),
ActiveNerf [PLSH22] does not always successfully choose the cameras that would
allow for a good reconstruction. This happens for many reasons. First, the
original NeRF models used has a hard time to converge in complicated scenarios
that are not similar to the synthetic blender dataset, and it becomes even
harder with the Bayesian model of Active-NeRF. Active-NeRF needs to get a
notion of the scene to allow for good camera placements and in complicated
scenes it can be challenging just from 20 initial cameras. Second, ActiveNeRF
chooses cameras that maximize the uncertainty for the specific model they are
training for, this does not guarantee that this uncertainty metric will
generalize to other NeRF models and finally the memory and speed requirements
do not allow for a huge number of candidate cameras similar to our method.

We can see that our capture style outperforms all other algorithms across the
scenes and views both quantitatively in Tab. 1 and qualitatively at Fig. 2 and
Fig. 5. This supports our hypothesis that if we observe all parts of the scene
while maintaining a uniform set of directions we will get an ideal
reconstruction.

We also perform a visual analysis to provide insight on how different methods
score against the energy function in Eq. (3). In Fig. 3 we provide a
visualization of the scores of each of the two terms of Eq. (3). We see that
our method clearly observes all the nodes more often that the other baselines
and we achieve better angular coverage for each node.

We also show in Fig. 4 how our camera sampling improves in the test-sets as we
introduce more and more cameras against the random cameras.

Figure 6:  We also do a prelimary simulated evaluation of our method with a
real scene, in which we captured approximately 1300 images, from which we
exclude every 14th image and create a test-set. We use our algorithm to select
the best 200 and we compared it against choosing 200 images at random. On
average Our selection scored 16.4 PSNR while the random selection scored 13.8
PSNR.

###  4.2 Preliminary Real Scene Evaluation

As discussed earlier (Sec. 3.4, we leave the actual integration of our method
into a full capture system as future work. Such a system would require either
a user interface (e.g., on a phone) or interfacing with a robotic capture
system (e.g., a drone). However, it is instructive to see how well our method
works on real data, so we present a very preliminary test on a real scene.
Since we are lacking a capture agent, we instead _simulate_ the ability to
select new cameras as a proof of concept. Specifically, we achieve this by
taking approximately 1300 photos (removing every 14th image to create a left-
out test set), simulating “random coverage” of the scene. We then use our
algorithm to select 200 cameras from this pool of randomly distributed images.
This is evidently a very preliminary test, but the results shown in Fig. 6
show that our method performs significantly better than random selection.

##  5 Conclusions

We presented an efficient method for selecting cameras for NeRF capture in
complicated environments, targeting free-viewpoint navigation. Our key
contributions are the introduction of the angular and coverage metrics, and
our fast optimization to propose the next best camera for NeRF reconstruction.
Our method outperforms baselines and one previous method in overall
perfomance; it is also faster than other methods and without significant
overhead over baseline methods. An important attribute of our solution is that
it is easily interpretable and can provide meaningful guidance and
understanding to users without requiring additional images. One other benefit
from the simplicity of our methods is that it could be adapted to vary the
importance of the scene spatially; we leave this as future work.

Our method is not without limitations. One issue is that we have not
investigated if our sampling is biased. If this is the case, no matter how
many cameras we sample, we might not reach a “perfect” reconstruction and
visual quality. Also, even though the method is efficient, it would benefit
from even faster performance allow truly interactive capture.

There are numerous possibilities for future work. From a theoretical
perspective, we are interested in studying other metrics of reconstruction
quality in a more extensive and complete fashion. We are also very excited
about the idea of integrating our approach in a mixed Augmented/Virtual
Reality (AR/VR) context: for example we can guide an on-site (AR) user to take
photos of a scene so that the remote VR user can very quickly be immersed in
the same environment. Using our method in the context of drone capture would
allow NeRF captures to be performed with high quality with little human
intervention, rendering the approach much more useful and easy-to-use.

## References

  * [BKA∗16] Bircher A., Kamel M., Alexis K., Oleynikova H., Siegwart R.:  Receding horizon" next-best-view" planner for 3d exploration.  In _2016 IEEE international conference on robotics and automation (ICRA)_ (2016), IEEE, pp. 1462–1468.
  * [BMT∗21] Barron J. T., Mildenhall B., Tancik M., Hedman P., Martin-Brualla R., Srinivasan P. P.:  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ (2021), pp. 5855–5864.
  * [BMV∗22] Barron J. T., Mildenhall B., Verbin D., Srinivasan P. P., Hedman P.:  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2022), pp. 5470–5479.
  * [Com18] Community B. O.:  _Blender - a 3D modelling and rendering package_.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.  URL: http://www.blender.org.
  * [CXG∗22] Chen A., Xu Z., Geiger A., Yu J., Su H.:  Tensorf: Tensorial radiance fields.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ (2022), Springer, pp. 333–350.
  * [DF09] Dunn E., Frahm J.-M.:  Next best view planning for active model improvement.  In _BMVC_ (2009), pp. 1–11.
  * [FKYT∗22] Fridovich-Keil S., Yu A., Tancik M., Chen Q., Recht B., Kanazawa A.:  Plenoxels: Radiance fields without neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2022), pp. 5501–5510.
  * [ISDS16] Isler S., Sabzevari R., Delmerico J., Scaramuzza D.:  An information gain formulation for active volumetric 3d reconstruction.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ (2016), IEEE, pp. 3477–3484.
  * [JTA21] Jain A., Tancik M., Abbeel P.:  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ (2021), pp. 5885–5894.
  * [KDSB22] Kulhánek J., Derner E., Sattler T., Babuška R.:  Viewformer: Nerf-free neural rendering from few images using transformers.  In _European Conference on Computer Vision_ (2022), Springer, pp. 198–216.
  * [LCW∗22] Lee S., Chen L., Wang J., Liniger A., Kumar S., Yu F.:  Uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields.  _IEEE Robotics and Automation Letters 7_ , 4 (2022), 12070–12077.
  * [LFR22] Liu C., Fischer M., Ritschel T.:  Learning to learn and sample brdfs.  _arXiv preprint arXiv:2210.03510_ (2022).
  * [Lum23] Lumai.com:  _Getting started with Luma AI: Guided Capture Mode_ , 2023.  URL: https://www.youtube.com/watch?v=JXb_a3ZIGnI.
  * [MESK22] Müller T., Evans A., Schied C., Keller A.:  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG) 41_ , 4 (2022), 1–15.
  * [MRFB16] Mostegel C., Rumpler M., Fraundorfer F., Bischof H.:  Uav-based autonomous image acquisition with multi-view stereo quality assurance by confidence prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ (2016), pp. 1–10.
  * [MST∗21] Mildenhall B., Srinivasan P. P., Tancik M., Barron J. T., Ramamoorthi R., Ng R.:  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM 65_ , 1 (2021), 99–106.
  * [PLSH22] Pan X., Lai Z., Song S., Huang G.:  Activenerf: Learning where to see with uncertainty estimation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ (2022), Springer, pp. 230–246.
  * [RZH∗23] Ran Y., Zeng J., He S., Chen J., Li L., Chen Y., Lee G., Ye Q.:  Neurar: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations.  _IEEE Robotics and Automation Letters_ (2023).
  * [SF16] Schönberger J. L., Frahm J.-M.:  Structure-from-Motion Revisited.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016).
  * [SMGH18] Smith N., Moehrle N., Goesele M., Heidrich W.:  Aerial path planning for urban scene reconstruction: A continuous optimization method and benchmark.
  * [SSC22] Sun C., Sun M., Chen H.-T.:  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2022), pp. 5459–5469.
  * [SZFP16] Schönberger J. L., Zheng E., Frahm J.-M., Pollefeys M.:  Pixelwise view selection for unstructured multi-view stereo.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14_ (2016), Springer, pp. 501–518.
  * [TTM∗22] Tewari A., Thies J., Mildenhall B., Srinivasan P., Tretschk E., Yifan W., Lassner C., Sitzmann V., Martin-Brualla R., Lombardi S., et al.:  Advances in neural rendering.  In _Computer Graphics Forum_ (2022), vol. 41, Wiley Online Library, pp. 703–735.
  * [XTS∗22] Xie Y., Takikawa T., Saito S., Litany O., Yan S., Khan N., Tombari F., Tompkin J., Sitzmann V., Sridhar S.:  Neural fields in visual computing and beyond.  In _Computer Graphics Forum_ (2022), vol. 41, Wiley Online Library, pp. 641–676.
  * [XXP∗22] Xu Q., Xu Z., Philip J., Bi S., Shu Z., Sunkavalli K., Neumann U.:  Point-nerf: Point-based neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2022), pp. 5438–5448.
  * [YYTK21] Yu A., Ye V., Tancik M., Kanazawa A.:  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2021), pp. 4578–4587.
  * [ZLR∗22] Zeng J., Li Y., Ran Y., Li S., Gao F., Li L., He S., Ye Q., et al.:  Efficient view path planning for autonomous implicit reconstruction.  _arXiv preprint arXiv:2209.13159_ (2022).
  * [ZZX∗22] Zhan H., Zheng J., Xu Y., Reid I., Rezatofighi H.:  Activermap: Radiance field for active mapping and planning.  _arXiv preprint arXiv:2211.12656_ (2022).