# ParticleNeRF: A Particle-Based Encoding
for Online Neural Radiance Fields

Jad Abou-Chakra 1 Feras Dayoub 2 Niko Sünderhauf 1 1 Queensland University of
Technology
2 University of Adelaide

###### Abstract

While existing Neural Radiance Fields (NeRFs) for dynamic scenes are offline
methods with an emphasis on visual fidelity, our paper addresses the online
use case that prioritises real-time adaptability. We present ParticleNeRF, a
new approach that dynamically adapts to changes in the scene geometry by
learning an up-to-date representation online, every 200 ms. ParticleNeRF
achieves this using a novel particle-based parametric encoding. We couple
features to particles in space and backpropagate the photometric
reconstruction loss into the particles’ position gradients, which are then
interpreted as velocity vectors. Governed by a lightweight physics system to
handle collisions, this lets the features move freely with the changing scene
geometry. We demonstrate ParticleNeRF on various dynamic scenes containing
translating, rotating, articulated, and deformable objects. ParticleNeRF is
the first online dynamic NeRF and achieves fast adaptability with better
visual fidelity than brute-force online InstantNGP and other baseline
approaches on dynamic scenes with online constraints. Videos of our system can
be found at our project website https://sites.google.com/view/particlenerf.

Figure 1:  Our particle encoding significantly improves the ability to adapt
to changing scenes when compared to online InstantNGP [20], the top-performing
parametric encoding. The particle encoding fills the space with particles at
random locations, associating each particle with a feature. Both the positions
of the particles and their features are optimized during training. As the
wheel spins, the particles move to maintain a low reconstruction error.

##  1 Introduction

Neural Radiance Fields [19] are 3D scene representations trained from images
using a differentiable rendering process that allows them to render scenes
from novel viewpoints with high visual fidelity.

Dynamic NeRFs [25, 26, 24, 23, 8, 6] learn to represent _dynamic_ scenes.
However, these methods are currently _offline_ , which means they require
access to the _entire_ image sequence during training and can take several
minutes or even hours to train for sequences that last only seconds in real-
time.

Our paper addresses the challenge of _online_ dynamic Neural Radiance Fields
that can continuously learn an up-to-date implicit 3D representation of the
dynamic scene in real-time, without the need for access to future frames
during training.

We introduce ParticleNeRF, a novel dynamic NeRF approach that uses a particle-
based encoding to represent the dynamic scene. Our key insight and novelty –
illustrated in Fig. 1 – is that backpropagating the reconstruction loss into
both the positions and the features of the particles enables the embedding
features to move in space as the scene changes. As we illustrate in Fig. 2,
this lets ParticleNeRF elegantly adapt to moving, articulated or deforming
objects by adjusting the position of the feature-carrying particles.

ParticleNeRF uses a simple but effective physics system to govern the
particles’ positions and velocities, interpreting the individual position
gradients as velocity vectors. The physics system implements collision
constraints that prevent the particles from accumulating too close to each
other. This creates a soft upper limit on the number of nearest neighbours to
be considered for interpolation when calculating the feature for a query
point, thereby supporting ParticleNeRF’s efficient training process.

In contrast to the time-conditioned deformation networks used in [25, 26, 24,
23, 6, 8], which are designed for _offline_ training, ParticleNeRF
incrementally adapts to changes in the environment every 200 ms – making it
suitable for _online_ usage.

ParticleNeRF is the first method to address the challenge of _online_ learning
of dynamic scenes (including rigid body transforms, articulated objects, and
deforming objects). Our experiments show that ParticleNeRF can represent
dynamic scenes with higher fidelity than (i) brute-force online InstantNGP
that simply continuously trains the network as new frames become available,
(ii) and other baselines (Section 5). Our ablation in Section 6 shows that
this superior performance on dynamic scenes is indeed due to the ability to
backpropagate into the positions and move the feature-carrying particles
during the training process. We furthermore show that ParticleNeRF’s
performance degrades gracefully with a decrease in the number of particles and
the nearest neighbour search radius.

Figure 2:  In a rigid grid-based encoding (e.g. InstantNGP [20]), features are
associated with fixed positions in space and cannot move. If the scene
geometry changes, features have to be unlearned and relearned in different
positions. In contrast, our particle encoding associates features with
particles that can move freely in 3D space. As the scene geometry changes, the
particles can move to accommodate the scene change, resulting in faster
adaptation and higher quality representation of dynamic scenes.

In summary, our contributions are:

  1. 1.

A novel online NeRF formulation that can adapt to the changes in dynamic
scenes every 200 ms without using time-conditioned deformation networks;

  2. 2.

A new particle-based encoding that associates features with moving particles
and allows the gradients from the photometric loss to propagate to and change
the positions of the particles;

  3. 3.

The incorporation of a physics system into the NeRF formulation to update the
motion of the particles while preventing collisions.

Our implementation will be made available upon publication. Videos of
ParticleNeRF rendering dynamic scenes can be found on our project website
https://sites.google.com/view/particlenerf.

##  2 Related Work

We review NeRFs in the context of (i) how they have been used to tackle
dynamic environments, (ii) how certain encodings allow close to realtime
training and rendering, and lastly (iii) how our particle encoding is
different to three other point-based encodings suggested to date.

#### Dynamic NeRFs

Several papers extended the NeRF model to encompass dynamic scenes [25, 26,
24, 23, 6, 8, 32, 9, 15]. The distinguishing factor between the static and
dynamic NeRF formulations is that the first represents a single static scene
while the latter encodes the evolution of a 3D scene over time. During
training, these dynamic NeRFs can access images taken at any time step. They
are therefore designed to be trained after a sequence of images has already
been taken. To date, the majority of these works are not close to being
realtime capable taking minutes and even hours to train. Since these dynamic
NeRF approaches use the complete set of images during training, we refer to
them as “offline” methods. Alternatively, we seek to create a NeRF at every
timestep and only capture the latest state of the scene. In this context, our
method is “online”. For many applications, representing a time-varying scene
within the NeRF itself is unnecessary. A robot, for example, is usually only
interested in the latest state of the world.

Learning an online NeRF is therefore the challenge of learning a NeRF at a
timestep $t$ given a NeRF at a timestep $t-1$ and a set of images taken at
time $t$. Our particle encoding is formulated to tackle this problem.

#### Embeddings

In general, a NeRF is composed of two main parts: the encoding, and a multi-
layered perceptron (MLP). An encoding transforms the spatial input of the NeRF
to an alternative feature space where learning can be made easier [29]. The
MLP transforms the resultant feature into values that represent color and
geometry. Mildenhall et al. [19] introduced the first version of NeRFs with a
Fourier encoding which is a simple trigonometric map. The Fourier encoding
does not have any parameters that are learnt as part of the training and is
not associated with a discrete datastructure. It is, therefore, referred to as
non-parametric, or functional. This version of NeRF takes hours to train
because it requires a large MLP to represent the scene. Later works [20, 27,
4] show that by storing a large set of features in a discrete datastructure,
the spatial input could be used to index into and interpolate between features
in that set. The computed features become the embeddings that are consumed by
the NeRF’s neural network. Since the features in the set are stored as
parameters and are updated during training, these are referred to as
“parametric” encodings by [20]. By storing features in memory, the
computational burden on the MLP is reduced and training speed is significantly
increased. The strategy used to index and interpolate into the feature set
distinguishes the various types of parametric encodings. Almost all the
parametric encodings conceptually rely on a fixed grid datastructure [20, 27,
4] with the exception of [33] which uses a point-based encoding. The grid
encompasses the workspace of the NeRF and each node within is associated with
a feature. Therefore, while the features associated with the nodes can be
learnt, they cannot be moved.

In this paper, we show that if the features are allowed to be both learnt and
moved, then NeRFs can be made to incrementally adapt to dynamic scenes. We
introduce a particle encoding, where features are associated with moving
points in space. We show that in a changing scene, the feature-holding
particles move in accordance with the scene – Figure 2.

#### Point-based NeRFs

There are three works that use points as part of their formulation. Point-NeRF
and SPIDR [33, 16] utilize points as a data structure to store features, but
their embedding differs significantly from ours. Notably, they do not address
the dynamic scene use case and they they do not allow their points to move
during optimization. In addition, they require depth data, their training
takes in the order of minutes to hours, and their encoding is not invariant to
$\text{SE}(3)$ transformations. NeuroFluid [11] uses particles and NeRFs to
specifically model the behaviour of fluids. Their encoding is non-parametric
and their points represent particles of a physical medium rather than
descriptors of an embedding space.

Figure 3:  Illustration of our particle encoding. (1) A query point
$\boldsymbol{x}$ is sampled in space. (2) The features and positions of the
particles within a search radius are retrieved. (3) The features and distances
from the query point are used to interpolate the feature at the query point
$\boldsymbol{x}$. (4) The resulting feature is evaluated by the neural network
to give color and density. To train the encoding, the loss gradients are
backpropagated through the network (4), the query feature (3), and finally
into the positions and features of the particles in (2).

##  3 NeRF Preliminaries

A NeRF is a continuous representation of a 3D scene that maps a point
$\boldsymbol{x}\in\mathbb{R}^{3}$ and a unit-norm view direction
$\boldsymbol{d}\in\mathbb{R}^{3}$ to a color $\boldsymbol{c}\in\mathbb{R}^{3}$
and a density value $\sigma\in\mathbb{R}$. During training, rays are cast from
camera centers through their image plane. Each ray $\boldsymbol{r}$ is
associated with a direction vector $\boldsymbol{d}$ and a set of points
$\boldsymbol{x}_{i}$ progressively sampled from the ray’s length [19]. The
points $\boldsymbol{x}_{i}$ and the direction vector $\boldsymbol{d}$ are
mapped by the NeRF to $\boldsymbol{c}_{i}$ and $\sigma_{i}$. Defining
$\delta_{i}=\lVert\boldsymbol{x}_{i+1}-\boldsymbol{x}_{i}\rVert$, the expected
color of the ray $\hat{C}(\boldsymbol{r})$ is given by:

$\hat{C}(\boldsymbol{r})=\sum_{i=1}^{N}w_{i}\boldsymbol{c}_{i}$ (1)

$w_{i}=T_{i}(1-\exp(-\sigma_{i}\delta_{i}))\quad\text{and}\quad
T_{i}=\exp{\left(-\sum_{j=1}^{i-1}{\sigma_{j}\delta_{j}}\right)}$

A photometric loss is defined as

$L_{\text{rgb}}=\sum_{\boldsymbol{r}\in\mathcal{R}}\|\hat{C}(\boldsymbol{r})-\boldsymbol{c}_{\text{gt}}(\boldsymbol{r})\|_{2}$
(2)

The ground-truth color $\boldsymbol{c}_{gt}(\boldsymbol{r})$ is the color of
the pixel in the training image that the ray $\boldsymbol{r}$ intersects.
$\mathcal{R}$ is the set of all rays that pass through the training images.
Refer to [28, 18] for an in-depth derivation.

#### Encodings

Given a point $\boldsymbol{x}_{i}$ and a direction vector
$\boldsymbol{d}_{i}$, a NeRF first maps each to an embedding space
$E_{x}(\boldsymbol{x}_{i})$ and $E_{d}(\boldsymbol{d}_{i})$. The embeddings
are subsequently consumed by a multilayered perceptron to output color and
density. Not including an embedding layer critically deteriorates ability of
the NeRF to reconstruct scenes [29]. The choice of the embedding layer can
significantly reduce the size of the MLP required [4, 20, 33] and in some
cases removes the need for one entirely [7]. Close to realtime performance is
possible due to a particular class of encodings that the authors of [20] refer
to as “parametric”. These encodings are characterized by a discrete
datastructure that can be queried by the incoming coordinate
$\boldsymbol{x}_{i}$ to produce a set of features $\mathbf{f}_{j}$. The
features $\mathbf{f}_{j}$ are then combined to produce an embedding
$\mathbf{f}_{i}$. By storing the features in memory, a large MLP is no longer
required and significant savings are made in both rendering and training times
when compared to other formulations that use functional encodings [19, 31, 2,
3, 5].

##  4 ParticleNeRF

ParticleNeRF introduces a new particle-based encoding that associates features
with moving particles and allows the gradients from the photometric loss to
propagate through and change the positions of the particles. This is a novel
method of dealing with dynamic scenes in an online manner without using time-
conditioned deformation networks.

Our particle encoding is formulated as a neural pointcloud
$\mathcal{P}=\\{(\boldsymbol{x}_{i},\boldsymbol{v}_{i},\mathbf{f}_{i}):i=1,2,...,M\\}$,
where $\boldsymbol{x}_{i}\in\mathbb{R}^{3}$ is the position of a latent
particle in the system, $\boldsymbol{v}_{i}\in\mathbb{R}^{3}$ is the velocity
of the particle, $\mathbf{f}_{i}\in\mathbb{R}^{m}$ is its associated feature,
and $M$ is the total number of particles. The pointcloud $\mathcal{P}$ is an
irregular grid of latent features that defines the map F from a coordinate
$\boldsymbol{x}_{j}\in\mathbb{R}^{3}$ to $\mathbf{f}_{j}$.

$\mathbf{f}_{j}=F(\boldsymbol{x}_{j},\mathcal{P})=\sum_{(\boldsymbol{x}_{i},\mathbf{f}_{i})\in\mathcal{P}}{w(\lVert\boldsymbol{x}_{j}-\boldsymbol{x}_{i}\rVert_{2})\mathbf{f}_{i}}$
(3)

where $w$ is the bump function – a compactly supported radial basis function
(RBF) – given by:

$w(r)=\begin{cases}\exp\left({-\displaystyle\frac{s^{2}}{s^{2}-r^{2}}}\right),&\text{for
}r\in(-s^{2},s^{2})\\\ 0,&\text{otherwise}\end{cases}$ (4)

The search radius $s$ controls the influence of a particle on a region of
space. The compact RBF decreases the influence of a given particle on the
interpolated feature to $0$ as its distance approaches the search radius. This
allows for the efficient calculation of $\mathbf{f}_{j}$ with a fixed-radius
nearest neighbour search algorithm [30] where the radius is set to $s$. We
find that using unnormalized features encourages the particles to spread
across the geometry.

ParticleNeRF uses the same architecture as InstantNGP (a 3 layer MLP with 64
neurons each), but replaces its hash encoding with our particle encoding. Note
that if a query point finds no neighbouring particles within the search radius
$s$, the feature is set to $\boldsymbol{0}$. Figure 3 illustrates our method.

We optimize the MLP parameters $\Phi$, the particle features $\mathbf{f}_{i}$
and the particle positions $\boldsymbol{x}_{i}$ relative to the NeRF loss
$L_{\text{rgb}}$ defined in Eq. 2:

$\Phi^{*},\\{\mathbf{f}_{i}\\}^{*},\\{\boldsymbol{x}_{i}\\}^{*}=\operatorname*{arg\,min}_{\Phi,\\{\mathbf{f}_{i}\\},\\{\boldsymbol{x}_{i}\\}}L_{\text{rgb}}$
(5)

A core novelty of our approach is that the gradients of the NeRF loss are used
to update _both_ the position and the associated features of the particles.
This allows the particles to move with the geometry in the scene as it
changes, leading to a higher fidelity representation of dynamic scenes. As our
experiments in Section 5 will show, ParticleNeRF can adapt to changes in the
scene _online_ , i.e. every 200 ms, which is orders of magnitude faster than
previous NeRFs in dynamic scenes [24, 23, 25, 32, 9, 15], and enables a much
higher reconstruction quality compared to online InstantNGP [20] and other
baselines.

###  4.1 Position Based Dynamics

We observe that propagating the NeRF loss into the particle positions can
sometimes lead to multiple particles accumulating in a small region of space.
Although this does not adversely affect the reconstruction quality, it is not
desirable because it needlessly increases the number of neighbours that each
query point has to process during training and evaluation. To address this
problem in a structured way, we build a position-based physics system [21, 17]
into InstantNGP that resolves the dynamics of our particles.

Position-based dynamics (PBD) is a simple, robust, and fast physics system
that can evolve the motion of our particles while resolving the constraints we
put on them. We interpret the gradients
$\frac{dL_{\text{rgb}}}{d\boldsymbol{x}_{i}}$ as scaled velocity vectors that
are added to the particles’ current velocity – Algorithm 1. Using PBD not only
provides a means of limiting the number of neighbours a particle has, but it
also creates future opportunities to add other constraints into the system.
For example, if an object is known to be a rigid object, or part of an
articulated body, the particles can be suitably constrained.

Algorithm 1 PBD Physics Step

1:for all particles $i$ do

2:
$\boldsymbol{v}_{i}\leftarrow\gamma\boldsymbol{v}_{i}-\alpha\displaystyle\frac{dL_{\text{rgb}}}{d\boldsymbol{x}_{i}}$
$\triangleright$ Update velocities

3: $\mathbf{p}_{i}\leftarrow\boldsymbol{x}_{i}$ $\triangleright$ Store
previous positions

4: $\boldsymbol{x}_{i}\leftarrow\boldsymbol{x}_{i}+\Delta t\boldsymbol{v}_{i}$
$\triangleright$ Integrate positions

5:for all collision pairs i, j do

6: $l\leftarrow||\boldsymbol{x}_{j}-\boldsymbol{x}_{i}||$

7:
$\boldsymbol{x}_{i}\leftarrow\boldsymbol{x}_{i}+0.5(l-\delta)\displaystyle\frac{\boldsymbol{x}_{j}-\boldsymbol{x}_{i}}{l}$
$\triangleright$ Resolve collisions

8:for all particles $i$ do

9: $\boldsymbol{v}_{i}\leftarrow(\boldsymbol{x}_{i}-\mathbf{p}_{i})/\Delta t$
$\triangleright$ Update velocities

In all our experiments, we choose a damping factor $\gamma=0.96$, a timestep
$\Delta t=0.01$, a minimum distance of $\delta=0.01$. The base InstantNGP
implementation transforms a given scene so that it fits within a unit cube.
Our minimum distance $\delta$ in the physics system and the search radius $s$
that is used to query neighbouring particles are given in the units of the
unit cube.

###  4.2 Implementation

We build our particle encoding as an extension of the InstantNGP
implementation released by Müller _et al_. [20]. We clip the particle position
gradients so that their norms do not exceed the search radius. We implement
the sorting-based fixed-radius nearest search algorithm described by [10] and
the position-based dynamics physics system described by [21] as additional
CUDA kernels [22] within InstantNGP and will make the implementation available
after publication. Two different Adam optimizers [14] are used. The first is
for the parameters of the MLP $\Phi$ and the second is for the particle
features $\mathbf{f}_{i}$. Both are parameterized by $\beta_{1}=0.9$,
$\beta_{2}=0.99$, $\epsilon=10^{-10}$ with the learning rate set at $0.01$.
The particle positions are updated through the PBD system where collision
constraints can be resolved.

##  5 Experiments

We experimentally evaluate ParticleNeRF and show its abilities to represent a
dynamic environment online.

Datasets: We construct a dynamic dataset consisting of 6 scenes, inspired by
D-NeRF [25], comprising of scenes with deformable and articulated moving
objects that loop. In addition, we created an animated version of the original
NeRF Blender dataset [19], by applying translations and rotations to the
objects.

We note that datasets such as [24, 23] that are commonly used to evaluate
offline dynamic NeRFs are not compatible with ParticleNeRF’s (and
InstantNGP’s) requirement for multiple cameras per frame. We discuss this
limitation further in Sec. 6.

Baseline: We compare ParticleNeRF against online InstantNGP [20] and online
TiNeuVox [6].

InstantNGP is the fastest NeRF implementation currently available, and it can
be utilized for online applications by continuously learning on new frames
without resetting the weights. We are not the first to propose this, as
EvoNeRF [12] uses a continuous data stream to train InstantNGP for a robotic
grasping task. InstantNGP’s fast training speed allows us to evaluate whether
a brute force approach of continuous learning is a feasible strategy for
online dynamic scenes.

As most of the dynamic NeRF approaches [24, 23, 25, 32, 9, 15] published to
date are offline methods that are not trained at the required speeds for
online applications, we do not compare our approach against them. However, a
recent offline method called TiNeuVox [6] has been released, which trains
within minutes by utilizing a time-conditioned deformation network and voxel-
based parametric encodings. We compare our approach against the two
configurations of TiNeuVox released by the authors: TiNeuVox and TiNeuVox-S.
We convert them to an online use-case where only images from the current
timestep can be sampled.

Hyperparameters: For all the experiments involving the particle encoding, we
initialize the unit cube containing the scene with a grid of uniformly spaced
particles (We obtain comparable outcomes when initializing the particles
randomly). Each particle is associated with a 4 dimensional feature and is
initialized randomly between $\pm 10^{-2}$. Increasing the dimension of the
feature has minimal impact.

Figure 4:  ParticleNeRF evaluated on 6 scenes with deformable, articulated,
and multiple object movements. Scenes are held static for 100 frames before
the animation begins. ParticleNeRF provides a stable representation during
movements. InstantNGP can recover when motions slow down. Online TiNeuVox
fails when movements get faster.  Figure 5:  Image captures from a novel
viewpoint at the 300th frame for ParticleNeRF, TiNeuVox, and InstantNGP.

###  5.1 Dynamic Dataset

We evaluate our method using our dynamic dataset (Figure 4). Each scene in the
dataset has one or multiple objects being looked upon by 20 cameras
distributed on the top half of the surface of a sphere. The objects are either
articulated (robot), deformable (spring, cloth), or rigid (wheel). The
remaining scenes are comprised of multiple moving objects (pendulums, robot
system).

During each experiment, the scene remains fixed for 500 training steps, after
which the animation starts. ParticleNeRF, online InstantNGP, and online
TiNeuVox have only five training iterations before loading the next animation
step.

At the end of every frame, we evaluate the photometric reconstruction quality
(SSIM, PSNR and LPIPS) from 10 unseen views. The SSIM values are plotted
against each frame in Figure 4. The average PSNR, SSIM, and PSNR values of all
the scenes for each method are shown in Table 1. Qualitative visualizations
are shown in Figure 5 and the full videos can be found on the project page.

Our findings reveal that ParticleNeRF effectively adapts to motion within a
small number of training steps, while the performance of online InstantNGP
noticeably declines under motion. InstantNGP is able to recover when the speed
of movements slow down. Nonetheless, ParticleNeRF exhibits reduced performance
when objects in the scene have varying sizes, which is due to its reliance on
a fixed search radius, as shown by the decreased performance in the robot
system scene. Additionally, online TiNeuVox exhibits artifacts, particularly
visible in the wheel scene, caused by the deformation network’s inability to
adapt within 5 training iterations.

Table 1: Average quantitative performance of methods on all dynamic scenes
Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
Particle | 27.47 | 0.94 | 0.08
InstantNGP | 24.69 | 0.91 | 0.12
TiNeuVox | 27.28 | 0.91 | 0.13
TiNeuVox-S | 26.64 | 0.92 | 0.14

###  5.2 Animated Blender Dataset

We further evaluate our method using a novel animated version of the standard
“Blender” dataset [19]. Each of the 8 scenes is made to rotate or translate at
a certain speed for 100 frames to test ParticleNeRF’s ability to deal with
elementary motions. At the end of every frame, we evaluate the photometric
reconstruction quality (PSNR) from 7 unseen views. In total, 56 animations are
tested and their results are summarized in Table 2. Videos and a more detailed
view of the results can also be found in the attached supplementary. The
results reinforce our findings that allowing particles to move provides a
method of adapting to moving geometries.

Table 2: Average PSNR on the animated blender scenes from [19].

Model |  Enc. | Rot. (∘/frm) | Trans. (cm/frm)
---|---|---|---
|  1 |  2 |  3 |  4 |  1 |  2 |  3
Chair |  ngp |  21 |  18 |  17 |  17 |  23 |  19 |  18
ours |  24 |  23 |  22 |  21 |  24 |  23 |  22
Drums |  ngp |  19 |  18 |  17 |  16 |  19 |  18 |  17
ours |  20 |  19 |  19 |  18 |  20 |  19 |  19
Ficus |  ngp |  22 |  22 |  21 |  21 |  22 |  21 |  21
ours |  23 |  22 |  22 |  21 |  22 |  22 |  21
Hotdog |  ngp |  26 |  24 |  23 |  23 |  25 |  23 |  21
ours |  27 |  27 |  26 |  25 |  26 |  25 |  24
Lego |  ngp |  21 |  19 |  18 |  18 |  21 |  20 |  19
ours |  23 |  22 |  21 |  21 |  23 |  22 |  21
Mic |  ngp |  26 |  24 |  23 |  23 |  27 |  24 |  23
ours |  28 |  27 |  26 |  25 |  28 |  27 |  26
Ship |  ngp |  23 |  22 |  22 |  21 |  22 |  21 |  21
ours |  23 |  23 |  23 |  22 |  23 |  23 |  22
Matrl. |  ngp |  21 |  19 |  17 |  17 |  23 |  21 |  19
ours |  23 |  23 |  22 |  21 |  23 |  23 |  23

|
---|---
(a) Rotation | (b) Translation

Figure 6:  On the Wheel dataset, ParticleNeRF maintains the structure of the
wheel as it rotates. We show how our particle encoding performs in three
settings: (i) When both the features and the positions of the particles are
optimized (blue). (ii) When only the positions of the particles are optimized
(yellow). (iii) When the only the particle features are optimized (green).
Lastly we show how InstantNGP [20] performs (orange). We illustrate the
results of the reconstruction from an unseen viewpoint at the frames indicated
for both InstantNGP and the complete ParticleNeRF, corresponding to setup (i).
Even at the lowest PSNR at frame 700, our method preserves the wheel’s
structure.  Figure 7:  An ablation showing the effect of the number of
particles (left) and the particle search radius (right) on the photometric
reconstruction quality. When ablating the number of particles, search radius
is set at 0.04. When ablating the search radius, number of particles is set at
100,000. In both cases, there is a diminishing effect at higher speeds.

###  5.3 Ablation

To thoroughly ablate and understand the dynamic representational ability of
ParticleNeRF, we use the wheel scene from the dynamic dataset. To investigate
ParticleNeRF’s and the baselines’ performance in static scenes and dynamic
scenes with varying degrees of motion, we change the rotation speed of the
wheel from static to successively faster speeds. Most of the following results
are illustrated in Fig. 6.

Models are given 5 training steps per frame and are configured with the same
parameters from Section 5.1.

#### Optimising Particle Positions

We first investigate the contribution of ParticleNeRF’s ability to
backpropagate the rendering loss into the particle positions to the overall
performance. In Fig. 6 we show PSNR performance when allowing the loss to
propagate into only the features (green); only the particle positions, keeping
the features at their random initialisations (yellow), and allowing to learn
both the positions and the features (blue). We make the following
observations: First, when allowing to lean only the features (green), the
performance is identical to InstantNGP (orange), especially at the higher
motion rates. This is not surprising, since in that setup ParticleNeRF
operates like InstantNGP with randomly positioned feature embeddings, instead
of a fixed grid structure. Second, keeping the features fixed, but allowing
the particles to move, leads to increased PSNR performance that already
outperfoems the baselines in all but a few dynamic frames. Third, the
combination of learning positions and features achieves best performance.

#### Increasing Motion Magnitude

From Fig. 6 we can see that brute-force InstantNGP achieves a higher PSNR on
static scenes, but drops significantly in dynamic scenes as it is not capable
of relearning features fast enough to adapt. While ParticleNeRF has a slightly
worse performance on static scenes, it maintains its PSNR with only slight
degradation as the scene becomes more dynamic. Comparing the renderings at the
bottom of Fig. 6, we can see the significant motion artifacts exhibited by
brute-force InstantNGP. We furthermore observe a ghosting effect where
InstantNGP quickly recovers structure upon the wheel’s return to its original
configuration. This is manifested as oscillations in the PSNR metric in Figure
6.

#### Influence of Search Radius

The right side of Figure 7 illustrates how increasing search radii have a
diminishing effect on reconstruction quality. We further observe that at the
smaller search radius of $0.03$ and a higher speed of $3.6^{\circ}$/frame,
reconstruction quality begins to degrade over time. This affect is due to a
divergence that occurs when the number neighbours around a particle drops
below the necessary level to accurately represent the underlying geometry. We
currently do not have a growing strategy that would detect such an event and
multiply the particles in that region. This edge case and future work for
implementing a growing strategy is further discussed in the supplementary.

#### Influence of Number of Particles

In Figure 7 (left), we observed that increasing the number of particles used
to initialize the scene asymptotically improves reconstruction quality, with
$300,000$ particles producing the same quality as $200,000$. However, we also
found that only approximately 7,000 particles contribute to regions with high
density, with the remaining particles providing minimal contribution. The
additional particles improve reconstruction quality in two ways: Firstly, they
enable more particles to begin near the geometry, thereby reducing the initial
convergence time. Secondly, as movement occurs, particles may be ejected from
the geometry, and having free particles in empty regions enables the geometry
to reabsorb these particles, thus maintaining reconstruction quality. These
observations suggest the need for future work on developing a pruning and
growing strategy for the particles.

##  6 Conclusions, Limitations and Future Work

Limitations Our particle embedding and online formulation require at least 10
views per frame to constrain the training, while offline dynamic methods that
use time-conditioned deformation networks can use frames from any point during
the trajectory to meet this requirement. We can overcome this disadvantage by
incorporating depth supervision [1] and additional regularizers into the NeRF
loss [13].

ParticleNeRF currently has a cubic memory requirement – albeit with a low
constant – which can be addressed by implementing a pruning strategy that
removes particles with a density below a certain threshold.

ParticleNeRF has lower visual fidelity on static scenes when compared to other
methods. While we prioritized creating an encoding that is invariant to rigid
transformations and quick to adapt to motions, we sacrificed the visual
fidelity that comes with either multilevel encodings or positional Fourier
transforms. These limitations are topics for future research.

Conclusion Our new particle-based parametric encoding, combined with the novel
ability to backpropagate the rendering loss into the particle positions,
enables fast adaptation to dynamic scenes. By leveraging the position of our
particles, we have shown an alternative approach to using time-conditioned
deformation networks for dynamic scenes. Our approach is quicker to adapt to
movement in the scenes providing a consistent representation that is robust to
movement.

## References

  * [1] Jad Abou-Chakra, Feras Dayoub, and Niko Sunderhauf.  Implicit object mapping with noisy data.  ArXiv, abs/2204.10516, 2022.
  * [2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [4] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In European Conference on Computer Vision (ECCV), 2022.
  * [5] Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey.  Garf: Gaussian activated radiance fields for high fidelity reconstruction and pose estimation.  arXiv e-prints, pages arXiv–2204, 2022.
  * [6] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian.  Fast dynamic radiance fields with time-aware neural voxels.  arxiv:2205.15285, 2022.
  * [7] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5501–5510, June 2022.
  * [8] Wanshui Gan, Hongbin Xu, Yi Huang, Shifeng Chen, and Naoto Yokoya.  V4d: Voxel for 4d novel view synthesis.  arXiv preprint arXiv:2205.14332, 2022.
  * [9] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5712–5721, October 2021.
  * [10] Simon Green.  Particle simulation using cuda.  NVIDIA whitepaper, 2010.
  * [11] Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang.  Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields.  In ICML, 2022.
  * [12] Justin Kerr, Letian Fu, Huang Huang, Jeffrey Ichnowski, Matthew Tancik, Yahav Avigal, Angjoo Kanazawa, and Ken Goldberg.  Evo-neRF: Evolving neRF for sequential robot grasping.  In 6th Annual Conference on Robot Learning, 2022.
  * [13] Mijeong Kim, Seonguk Seo, and Bohyung Han.  Infonerf: Ray entropy minimization for few-shot neural volume rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12912–12921, 2022.
  * [14] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [15] Tianye Li, Mira Slavcheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv.  Neural 3d video synthesis from multi-view video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5521–5531, June 2022.
  * [16] Ruofan Liang, Jiahao Zhang, Haoda Li, Chen Yang, and Nandita Vijaykumar.  Spidr: Sdf-based neural point fields for illumination and deformation.  arXiv preprint arXiv:2210.08398, 2022.
  * [17] Miles Macklin, Matthias Müller, and Nuttapong Chentanez.  Xpbd: Position-based simulation of compliant constrained dynamics.  In Proceedings of the 9th International Conference on Motion in Games, MIG ’16, page 49–54, New York, NY, USA, 2016. Association for Computing Machinery.
  * [18] N. Max.  Optical models for direct volume rendering.  IEEE Transactions on Visualization and Computer Graphics, 1(2):99–108, 1995.
  * [19] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [20] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, July 2022.
  * [21] Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff.  Position based dynamics.  Journal of Visual Communication and Image Representation, 18(2):109–118, 2007.
  * [22] NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek.  Cuda, release: 10.2.89, 2020.
  * [23] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  ICCV, 2021.
  * [24] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  ACM Trans. Graph., 40(6), dec 2021.
  * [25] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  arXiv preprint arXiv:2011.13961, 2020.
  * [26] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger.  Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields.  IEEE Transactions on Visualization and Computer Graphics, 2023.
  * [27] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459–5469, 2022.
  * [28] Andrea Tagliasacchi and Ben Mildenhall.  Volume rendering digest (for nerf).  arXiv preprint arXiv:2209.02417, 2022.
  * [29] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  NeurIPS, 2020.
  * [30] Matthias Teschner, Bruno Heidelberger, Matthias Müller, Danat Pomeranets, and Markus Gross.  Optimized spatial hashing for collision detection of deformable objects.  VMV’03: Proceedings of the Vision, Modeling, Visualization, 3, 12 2003.
  * [31] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan.  Ref-NeRF: Structured view-dependent appearance for neural radiance fields.  CVPR, 2022.
  * [32] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim.  Space-time neural irradiance fields for free-viewpoint video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9421–9431, June 2021.
  * [33] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-nerf: Point-based neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5438–5448, 2022.

Supplemental Materials: ParticleNeRF

##  S-I Animated Blender Dataset

We show a more detailed view of our results on the animated Blender dataset in
Table S1. Visuals can found in Figure S1.

##  S-II Ablation Studies

Physics vs Adam In Figure S3, we update the particle positions on the wheel
dataset using Adam instead of using the PBD physics system. Our results show
that the physics system produces higher quality reconstructions in dynamic
scenes when compared to Adam.

Selecting the Gradient Scale Figure S4 shows how the gradient scale $\alpha$ –
used to integrate the positions gradients into the particle velocities – was
chosen for the wheel dataset. A small $\alpha$ creates a noticeable drop in
quality when the wheel begins to rotate at frame 100 because the particle
velocities take longer to adjust to the required speed. A large gradient scale
creates particle instability.

Long Running Scene In Figure S5, the wheel experiment is run for around 2000
frames at $2.4^{\circ}\text{/second}$. We observe a degradation in quality
over time as a result of particles slowly losing neighbours during the
movement. When particles lose all their neighbours, a degenerate case occurs
which inhibits other particles from reassembling within each other’s search
radius. This can be addressed in future work by creating a particle growing
strategy that detects this case and does not allow lone particles to exist.
Note, however, that even at the lowest reconstruction quality, the wheel
maintains its shape.

Full Scene We also visually demonstrate in Figure S6 that ParticleNeRF can be
used to reconstruct a full scene – albeit at a reduced reconstruction quality
when compared to InstantNGP.

##  S-III Implementation Details

InstantNGP uses an occupancy grid as an acceleration structure to skip
evaluating points in free space. In the original implementation, this
structure was updated every 16 training steps by evaluating the density at
each vertex in the grid. This worked well for static scenes because it was
expected that the densities of the points would converge to their values and
not change thereafter. In our dynamic scenes, we update the occupancy grid at
every training step for both InstantNGP and ParticleNeRF. This allows the
acceleration structure to quickly change with the scene and therefore does not
inhibit training in areas which were once unoccupied but have since become
occupied.

Towards Realtime In our dynamic dataset, out of the 100,000 particles
initialized in the scene, only around 7000 are actually contributing to the
geometry of the object. If we prune away particles with a density less than a
certain threshold, our system can run a single training step in under 6 ms. A
pruning strategy however must be paired with a growing strategy so that
particles can appear again when they are needed. A simple implementation of a
growing strategy would be to replicate particles with less than a certain
number of neighbours. The implementation of a robust pruning and growing
strategy is left for future work.

The computational cost of updating the acceleration structure at each training
step is high, particularly for a ParticleNeRF with a large number of
particles, as shown in Figure S7. To address this issue, an alternative
acceleration structure that exploits the point-based nature of the encoding
can be utilized. Z-buffering particles is another acceleration structure that
can be used to skip free space. When a pruning strategy is implemented, it can
be assumed that only the particles that produce geometry are present in the
system. Therefore, Z-buffering can skip free space without requiring
additional evaluations of the MLP, which is currently necessary.

Finally, we observe that the backward pass takes 5 times longer than the
forward pass despite having a similar number of computations. The difference
is due to the memory contention caused by multiple query points trying to
update the same set of neighbouring particles. This too can be alleviated in
future work.

Figure S1:  We test our encoding on an animated version of the Blender
dataset. (a) shows an object rotating around its up axis. (b) shows an object
translating from side to side. InstantNGP cannot learn features fast enough to
maintain a high quality reconstruction. ParticleNeRF is able to move its
features in space and maintain the structure of the object.  Table S1:
Performance of InstantNGP and ParticleNerf on the Animated Blender Dataset
reported through the mean and standard deviation of the photometric PSNR over
100 frames.

white  Model Chair Drums Ficus Hotdog Lego Materials Mic Ship Step per Frame
Encoding white white Hash 27.43 21.82 26.40 30.13 27.00 25.25 29.61 22.24
Static Particle 24.60 19.93 22.92 27.35 23.23 23.33 28.06 23.09 white white
Hash 21.30 $\pm$ 2.3 18.72 $\pm$ 1.4 22.44 $\pm$ 0.9 25.71 $\pm$ 1.1 21.07
$\pm$ 1.4 21.28 $\pm$ 1.5 26.42 $\pm$ 1.0 22.71 $\pm$ 1.1 white 1∘ Particle
24.02 $\pm$ 1.4 19.70 $\pm$ 1.2 22.85 $\pm$ 0.7 27.02 $\pm$ 0.8 22.77 $\pm$
1.1 23.23 $\pm$ 1.2 27.77 $\pm$ 0.5 23.04 $\pm$ 1.0 white white Hash 18.20
$\pm$ 1.9 17.76 $\pm$ 1.3 21.80 $\pm$ 0.9 24.13 $\pm$ 1.2 19.25 $\pm$ 1.4
18.80 $\pm$ 1.5 24.07 $\pm$ 1.0 22.15 $\pm$ 1.2 white 2∘ Particle 23.02 $\pm$
1.4 19.26 $\pm$ 1.2 22.23 $\pm$ 0.7 26.83 $\pm$ 0.8 22.01 $\pm$ 1.1 22.65
$\pm$ 1.2 26.97 $\pm$ 0.5 22.93 $\pm$ 1.0 white white Hash 16.91 $\pm$ 1.7
16.92 $\pm$ 1.2 21.19 $\pm$ 0.9 23.22 $\pm$ 1.3 18.29 $\pm$ 1.5 17.20 $\pm$
1.5 22.76 $\pm$ 1.4 21.64 $\pm$ 1.4 white 3∘ Particle 21.94 $\pm$ 1.5 18.72
$\pm$ 1.3 21.77 $\pm$ 0.8 26.15 $\pm$ 0.8 21.32 $\pm$ 1.1 22.11 $\pm$ 1.2
26.32 $\pm$ 0.6 22.64 $\pm$ 1.0 white white Hash 16.56 $\pm$ 1.8 16.38 $\pm$
1.0 20.66 $\pm$ 1.0 22.77 $\pm$ 1.4 17.65 $\pm$ 1.7 16.72 $\pm$ 1.5 22.99
$\pm$ 1.1 21.53 $\pm$ 1.3 white  Rotation 4∘ Particle 21.16 $\pm$ 1.5 18.26
$\pm$ 1.2 21.13 $\pm$ 0.8 25.45 $\pm$ 0.9 20.73 $\pm$ 1.2 21.38 $\pm$ 1.3
25.28 $\pm$ 0.7 22.21 $\pm$ 1.1 white white Hash 22.60 $\pm$ 2.1 19.06 $\pm$
1.5 22.13 $\pm$ 0.8 25.18 $\pm$ 1.7 21.40 $\pm$ 1.6 22.87 $\pm$ 1.6 27.06
$\pm$ 1.1 21.89 $\pm$ 1.4 white $1\,\text{cm}$ Particle 24.20 $\pm$ 1.4 19.68
$\pm$ 1.3 22.43 $\pm$ 0.7 26.11 $\pm$ 1.5 22.85 $\pm$ 1.1 23.34 $\pm$ 1.3
28.03 $\pm$ 0.6 22.69 $\pm$ 1.2 white white Hash 19.24 $\pm$ 2.1 17.68 $\pm$
1.4 21.38 $\pm$ 0.9 22.97 $\pm$ 1.7 20.06 $\pm$ 1.6 21.07 $\pm$ 1.9 24.43
$\pm$ 1.3 21.41 $\pm$ 1.3 white $2\,\text{cm}$ Particle 23.15 $\pm$ 1.3 19.31
$\pm$ 1.3 21.60 $\pm$ 0.7 25.38 $\pm$ 1.2 22.07 $\pm$ 1.1 23.06 $\pm$ 1.2
27.12 $\pm$ 0.7 22.71 $\pm$ 1.1 white white Hash 17.90 $\pm$ 2.0 17.12 $\pm$
1.3 21.11 $\pm$ 0.9 21.45 $\pm$ 1.8 19.37 $\pm$ 1.8 19.18 $\pm$ 2.3 23.47
$\pm$ 1.4 21.07 $\pm$ 1.4 white  Translation $3\,\text{cm}$ Particle 22.15
$\pm$ 1.4 18.80 $\pm$ 1.2 20.94 $\pm$ 0.7 24.28 $\pm$ 1.3 21.29 $\pm$ 1.2
22.69 $\pm$ 1.2 26.16 $\pm$ 0.7 22.15 $\pm$ 1.2

##  S-IV $\text{SE}(3)$ Invariance and Interpretability

Our encoding is invariant under 3D rigid transformations. For a transformation
$T\in\text{SE}(3)$:

$\mathbf{f}_{j}=F(\boldsymbol{x}_{j},\mathcal{P})=F(T\boldsymbol{x}_{j},T\circ\mathcal{P})$
(S1)

where

$T\circ\mathcal{P}=\\{(T\boldsymbol{x}_{i},T\boldsymbol{v}_{i},\mathbf{f}_{i}):i=1,2,...,M\\}$
(S2)

Our particle encoding approach generates local features that are directly
associated with local geometry patches. As a result of its $\text{SE}(3)$
invariance, when these particles are moved or deformed, the underlying
geometry will be similarly affected. For instance, if a subset of particles
representing an object within a scene is moved or deformed, the corresponding
geometry will change in predictable ways. In contrast, non-parametric NeRF
methods encode geometry within the weights of a neural network. There is no
natural transformation that can be applied to the weights that would change
the scene in predictable ways. Additionally, voxel-based features,
particularly those using a multilevel approach, cannot be easily transformed
to modify the scene. The lack of controllability and interpretability over
these features creates two challenges: (i) It is challenging to apply motion
priors in dynamic scenes - for example in the case where a certain object’s
velocity is known. This limitation was also identified by the authors of
TiNeuVox. (ii) The usefulness of NeRFs as a representation of geometry is
limited when the underlying features lack the invariance property. Therefore,
NeRFs without the invariance property are primarily suitable for view
synthesis and not as a new method of geometry representation in downstream
tasks.

Figure S2:  A degenerate case occurs when none of the particles have any
neighbours within their search radius. The resulting reconstruction manifests
as a set of spheres due to the inability of one particle alone to express more
complicated geometries.

##  S-V Degenerate Case

The interpolated features around a lone particle will be the same along the
surface of a sphere centered on that particle. The effect is shown visually in
Figure S2 where we reduce the number of particles in the scene to 200 and
ensure that every particle has no neighbours within its search radius. In our
experiments, we used a sufficient number of particles when initializing the
scene to circumvent the issue. Future work can develop a particle growing
algorithm that detects lone particles and creates neighbours in their
vicinity.

Figure S3:  A comparison between using Adam and the physics system to update
the particle positions after calculating their gradients relative to the NeRF
reconstruction loss. Integrating the gradients with the physics system
produces higher quality reconstructions on the wheel dataset whilst providing
a well formulated means of adding constraints. Adam is configured with
$\beta_{1}=0.9$, $\beta_{2}=0.99$ and a learning rate which is indicated in
the legend.  Figure S4:  An ablation performed on the wheel dataset with
100,000 particles and a search radius of 0.04 showing the impact of the
gradient scale $\alpha$ used to combine the NeRF gradients with the particle
velocities. Values around 2.0 have the best results.  Figure S5:  Figure
showing degradation that occurs over a longer sequence of the Wheel dataset.
The initial loss in quality is because the transition from static to 2.4
degrees is more sudden than the static to 1.2 degrees used in earlier
experiments. Degradation occurs as particles begin to lose neighbours and are
unable to acquire new ones.  Figure S6:  ParticleNeRF trained on a full scene
after 3000 training steps with 200,000 particles at a search radius of 0.02.
Figure S7:  We profile ParticleNeRF and InstantNGP and show where the majority
of the training time is spent per training step. The time taken by the
acceleration structure (Grid), the forward pass, the backward pass, and the
physics system are recorded for a range of configurations. InstantNGP is also
shown on the right.


$\mathcal{X}_{\sigma}$ – and $W^{\prime}\leq W$. Thus $W^{\prime}$ is also
$\leq$-minimal in $\mathcal{X}_{\sigma}$, i.e.
$W^{\prime}\in\mathcal{Y}_{\sigma}$. Since $W^{\prime}\in\mathcal{W}_{c\ :\
v}$ also, we have by Lemma 2 that
$v\in\operatorname{mod}_{0}\left[{B^{\sigma}_{c}}\right]$, as required.

“only if”: Suppose our operator is selective, i.e. satisfies _Boundedness_. To
show the desired property holds, take any $W$, $c$ and $v$. Enumerate
$\mathcal{C}\setminus\\{c\\}$ as $\\{d_{1},\ldots,d_{N}\\}$. By Lemma 7, for
each $1\leq n\leq N$ there is a $\ast$-consistent sequence $\sigma_{n}$ such
that

$\operatorname{mod}(G^{\sigma_{n}}_{\mathsf{snd}})=\\{W^{\prime}\in\mathcal{W}\mid
W\text{ refines }W^{\prime}\text{ at }d_{n}\\}.$

Now, let $\varphi$ and $\psi$ be formulas with
$\operatorname{mod}_{0}(\varphi)=\\{v\\}$ and
$\operatorname{mod}_{0}(\psi)=\\{v^{W}_{c}\\}$. Let $\rho$ be the
concatenation

$\rho=\sigma_{1}\cdots\sigma_{n}\cdot\langle{\ast,c,\varphi\lor\psi}\rangle.$

Note that $\rho$ is $\ast$-consistent, since each $\sigma_{n}$ is (and only
refers to case $d_{n}$). We may therefore apply _Boundedness_ for case $c$.
Taking models of both sides yields

$\operatorname{mod}_{0}\left[{B^{\rho}_{c}}\right]\supseteq\operatorname{mod}_{0}(\Gamma^{\rho}_{c})=\operatorname{mod}_{0}(\varphi\lor\psi)=\\{v,v^{W}_{c}\\}.$

In particular, $v\in\operatorname{mod}_{0}\left[{B^{\rho}_{c}}\right]$. By
Lemma 2, there is some $W^{\prime}\in\mathcal{Y}_{\rho}\cap\mathcal{W}_{c\ :\
v}$.

We show $W^{\prime}$ has the required properties. First note that since $W$
refines itself at each $d_{n}$, we have
$W\in\operatorname{mod}(G^{\sigma_{n}}_{\mathsf{snd}})$. Clearly
$W,c\models\psi$, so $W,c\models S_{\ast}(\varphi\lor\psi)$ too. Thus
$W\in\operatorname{mod}(G^{\rho}_{\mathsf{snd}})=\mathcal{X}_{\rho}$ (using
$K^{\emptyset}=\operatorname{Cn}(\emptyset)$). Since
$W^{\prime}\in\mathcal{Y}_{\rho}=\min_{\leq}{\mathcal{X}_{\rho}}$, we get
$W^{\prime}\leq W$ as required.

Next, take any case $d\neq c$. Then there is some $n$ such that $d=d_{n}$.
Since
$W^{\prime}\in\mathcal{Y}_{\rho}\subseteq\mathcal{X}_{\rho}=\operatorname{mod}(G^{\rho}_{\mathsf{snd}})\subseteq\operatorname{mod}(G^{\sigma_{n}}_{\mathsf{snd}})$,
we get that $W$ refines $W^{\prime}$ at $d$. This completes the proof.

∎

###  A.15 Proof of Theorem 4

###### Proof.

“only if”: Suppose a model-based operator is selective according to some case-
independent scheme $f$. Take any $\ast$-consistent $\sigma$,
$H\subseteq\mathcal{C}$ and $c\in\mathcal{C}$. For any case $d$, write
$M_{d}=\operatorname{mod}_{0}\left[{B^{\sigma}_{d}}\right]$. Note that with
$c_{0}$ an arbitrary fixed case, and writing
$F_{\sigma}(i,\varphi)=\operatorname{mod}_{0}(f_{\sigma}(i,c_{0},\varphi))$,
we have by case-independent-selectivity that

$M_{d}=\bigcap_{\langle{i,\varphi}\rangle\in\sigma\upharpoonright
d}{F_{\sigma}(i,\varphi)}.$

By closure, it is sufficient for _H-Boundedness_ to show that

$M_{c}\supseteq\operatorname{mod}_{0}(\Gamma^{\sigma,H}_{c})\cap\bigcap_{d\in
H}{M_{d}}.$ (1)

Take any $v$ in the set on the right-hand side. To show $v\in M_{c}$, take any
$\langle{i,\varphi}\rangle\in\sigma\upharpoonright c$. If
$\varphi\in\Gamma^{\sigma,H}_{c}$, then clearly

$\displaystyle v$
$\displaystyle\in\operatorname{mod}_{0}(\Gamma^{\sigma,H}_{c})$
$\displaystyle\subseteq\operatorname{mod}_{0}(\varphi)$
$\displaystyle\subseteq\operatorname{mod}_{0}(f_{\sigma}(i,c,\varphi))$
$\displaystyle=F_{\sigma}(i,\varphi)$

(where we use $f_{\sigma}(i,c,\varphi)\in\operatorname{Cn}_{0}(\varphi)$).
Otherwise, $\varphi\notin\Gamma^{\sigma,H}_{c}$. Since
$\langle{i,\varphi}\rangle\in\sigma\upharpoonright c$, this means there is
$d\in H$ such that $\langle{i,\varphi}\rangle\in\sigma\upharpoonright d$.
Hence $v\in M_{d}$ gives $v\in F_{\sigma}(i,\varphi)$. This shows the
inclusion in 1, and we are done.

“if”: Suppose a model-based operator satisfies _H-Boundedness_. Let $\sigma$
be a $\ast$-consistent sequence. As before, write $M_{c}$ for
$\operatorname{mod}_{0}\left[{B^{\sigma}_{c}}\right]$. For $i\in\mathcal{S}$
and $c\in\mathcal{C}$, write

$\mathcal{C}(i,\varphi)=\\{c\in\mathcal{C}\mid\langle{i,\varphi}\rangle\in\sigma\upharpoonright
c\\},$

and set

$F_{\sigma}(i,\varphi)=\operatorname{mod}_{0}(\varphi)\cup\bigcup_{c\in\mathcal{C}(i,\varphi)}{M_{c}}.$

Define $f$ by letting $f_{\sigma}(i,c,\varphi)$ be any propositional formula
with $\operatorname{mod}_{0}(f_{\sigma}(i,c,\varphi))=F_{\sigma}(i,\varphi)$.
Then $f$ is a case-independent selection scheme. We show our operator is
selective according to $f$; by closure of $\left[{B^{\sigma}_{c}}\right]$ for
each $c$, it suffices to show

$M_{c}=\bigcap_{\langle{i,\varphi}\rangle\in\sigma\upharpoonright
c}{F_{\sigma}(i,\varphi)}.$

Fix $c$. For the left-to-right inclusion, suppose $v\in M_{c}$. Take any
$\langle{i,\varphi}\rangle\in\sigma\upharpoonright c$. Then
$c\in\mathcal{C}(i,\varphi)$, so $F_{\sigma}(i,\varphi)\supseteq M_{c}$ and
thus $v\in F_{\sigma}(i,\varphi)$ as required.

For the right-to-left inclusion, suppose $v$ lies in the intersection. Set

$H=\\{d\in\mathcal{C}\mid v\in M_{d}\\}.$

Apply _H-Boundedness_ and taking the models of both sides, we obtain

$M_{c}\supseteq\operatorname{mod}_{0}(\Gamma^{\sigma,H}_{c})\cap\bigcap_{d\in
H}{M_{d}}.$ (2)

Clearly $v\in\bigcap_{d\in H}{M_{d}}$ by definition of $H$. Let
$\varphi\in\Gamma^{\sigma,H}_{c}$. Then there is $i\in\mathcal{S}$ such that
$\langle{i,\varphi}\rangle\in\sigma\upharpoonright c$, and consequently $v\in
F_{\sigma}(i,\varphi)$. We claim $v\in\operatorname{mod}_{0}(\varphi)$. If
not, by definition of $F_{\sigma}(i,\varphi)$ we must have
$v\in\bigcup_{d\in\mathcal{C}(i,\varphi)}{M_{d}}$, i.e. there is
$d\in\mathcal{C}$ such that $\langle{i,\varphi}\rangle\in\sigma\upharpoonright
d$ and $v\in M_{d}$. On the one hand, $\varphi\in\Gamma^{\sigma,H}_{c}$
implies $d\notin H$. On the other, $v\in M_{d}$ gives $d\in H$ directly by the
definition of $H$: contradiction. This shows
$v\in\operatorname{mod}_{0}(\varphi)$. Since $\varphi$ was arbitrary, we have
$v\in\operatorname{mod}_{0}(\Gamma^{\sigma,H}_{c})$. By 2 we get $v\in M_{c}$,
and the proof is complete.

∎

## References

  * Alchourrón, Gärdenfors, and Makinson 1985 Alchourrón, C. E.; Gärdenfors, P.; and Makinson, D.  1985\.  On the logic of theory change: Partial meet contraction and revision functions.  Journal of symbolic logic 510–530.
  * Booth and Hunter 2018 Booth, R., and Hunter, A.  2018\.  Trust as a Precursor to Belief Revision.  J. Artif. Intell. Res. 61:699–722.
  * Delgrande, Dubois, and Lang 2006 Delgrande, J. P.; Dubois, D.; and Lang, J.  2006\.  Iterated revision as prioritized merging.  KR 6:210–220.
  * Delgrande, Peppas, and Woltran 2018 Delgrande, J. P.; Peppas, P.; and Woltran, S.  2018\.  General belief revision.  Journal of the ACM (JACM) 65(5):1–34.
  * Fermé and Hansson 1999 Fermé, E. L., and Hansson, S. O.  1999\.  Selective revision.  Studia Logica 63(3):331–342.
  * Grove 1988 Grove, A.  1988\.  Two modellings for theory change.  Journal of philosophical logic 157–170.
  * Grégoire and Konieczny 2006 Grégoire, E., and Konieczny, S.  2006\.  Logic-based approaches to information fusion.  Information Fusion 7(1):4 – 18.  Logic-based Approaches to Information Fusion.
  * Hansson et al. 2001 Hansson, S. O.; Fermé, E. L.; Cantwell, J.; and Falappa, M. A.  2001\.  Credibility limited revision.  Journal of Symbolic Logic 66(4):1581–1596.
  * Hansson 1999 Hansson, S. O.  1999\.  A survey of non-prioritized belief revision.  Erkenntnis 50(2):413–427.
  * Hunter 2021 Hunter, A.  2021\.  Building trust for belief revision.  In Pham, D. N.; Theeramunkong, T.; Governatori, G.; and Liu, F., eds., PRICAI 2021: Trends in Artificial Intelligence, 543–555.  Cham: Springer International Publishing.
  * Katsuno and Mendelzon 1991 Katsuno, H., and Mendelzon, A. O.  1991\.  Propositional knowledge base revision and minimal change.  Artificial Intelligence 52(3):263–294.
  * Konieczny and Pino Pérez 2002 Konieczny, S., and Pino Pérez, R.  2002\.  Merging information under constraints: a logical framework.  Journal of Logic and Computation 12(5):773–808.
  * Kraus, Lehmann, and Magidor 1990 Kraus, S.; Lehmann, D.; and Magidor, M.  1990\.  Nonmonotonic reasoning, preferential models and cumulative logics.  Artificial intelligence 44(1-2):167–207.
  * Li et al. 2016 Li, Y.; Gao, J.; Meng, C.; Li, Q.; Su, L.; Zhao, B.; Fan, W.; and Han, J.  2016\.  A Survey on Truth Discovery.  SIGKDD Explor. Newsl. 17(2):1–16.
  * Schwind and Konieczny 2020 Schwind, N., and Konieczny, S.  2020\.  Non-Prioritized Iterated Revision: Improvement via Incremental Belief Merging.  In Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, 738–747.
  * Singleton 2021 Singleton, J.  2021\.  A logic of expertise.  ESSLLI 2021 Student Session.  https://arxiv.org/abs/2107.10832.
  * van Ditmarsch et al. 2015 van Ditmarsch, H.; van der Hoek, W.; Halpern, J.; and Kooi, B., eds.  2015\.  Handbook of Epistemic Logic.  College Publications.
  * Yasser and Ismail 2021 Yasser, A., and Ismail, H. O.  2021\.  Trust is all you need: From belief revision to information revision.  In Faber, W.; Friedrich, G.; Gebser, M.; and Morak, M., eds., Logics in Artificial Intelligence, 50–65.  Cham: Springer International Publishing.


#  Embedding Theory of Reservoir Computing and Reducing Reservoir Network
Using Time Delays

Xing-Yue Duan  School of Mathematical Sciences, Soochow University, Suzhou
215006, China  Xiong Ying  School of Mathematical Sciences, SCMS, and SCAM,
Fudan University, Shanghai 200433, China  Si-Yang Leng  Research Institute of
Intelligent Complex Systems and Centre for Computational Systems Biology,
Fudan University, Shanghai 200433, China  Academy for Engineering and
Technology, Fudan University, Shanghai 200433, China  State Key Laboratory of
Medical Neurobiology, LCNBI, and MOE Frontiers Center for Brain Science,
Institutes of Brain Science, Fudan University, Shanghai 200032, China  Jürgen
Kurths  Research Institute of Intelligent Complex Systems and Centre for
Computational Systems Biology, Fudan University, Shanghai 200433, China
Potsdam Institute for Climate Impact Research (PIK), Potsdam 14473, Germany
Wei Lin<EMAIL_ADDRESS>School of Mathematical Sciences, SCMS, and SCAM,
Fudan University, Shanghai 200433, China  Research Institute of Intelligent
Complex Systems and Centre for Computational Systems Biology, Fudan
University, Shanghai 200433, China  State Key Laboratory of Medical
Neurobiology, LCNBI, and MOE Frontiers Center for Brain Science, Institutes of
Brain Science, Fudan University, Shanghai 200032, China  Shanghai Artificial
Intelligence Laboratory, Shanghai 200232, China  Huan-Fei Ma<EMAIL_ADDRESS>School of Mathematical Sciences, Soochow University, Suzhou 215006, China

###### Abstract

Reservoir computing (RC), a particular form of recurrent neural network, is
under explosive development due to its exceptional efficacy and high
performance in reconstruction or/and prediction of complex physical systems.
However, the mechanism triggering such effective applications of RC is still
unclear, awaiting deep and systematic exploration. Here, combining the delayed
embedding theory with the generalized embedding theory, we rigorously prove
that RC is essentially a high dimensional embedding of the original input
nonlinear dynamical system. Thus, using this embedding property, we unify into
a universal framework the standard RC and the time-delayed RC where we novelly
introduce time delays only into the network’s output layer, and we further
find a trade-off relation between the time delays and the number of neurons in
RC. Based on these findings, we significantly reduce the RC’s network size and
promote its memory capacity in completing systems reconstruction and
prediction. More surprisingly, only using a single-neuron reservoir with time
delays is sometimes sufficient for achieving reconstruction and prediction
tasks, while the standard RC of any large size but without time delay cannot
complete them yet.

###### pacs:

The last decades have witnessed the extensive application and development of
machine learning technology in data-driven research and in high-technology-
oriented industry as well. As a representative leader among many machine
learning techniques, the artificial neural network (ANN) has emerged as a
powerful approach that is well suited for coping with the supervised learning
problems. Among various architectures of ANN, Reservoir Computing (RC), which
is a recently developed framework [1], a special variant of a recurrent neural
network, and also known as a generalization of echo-state network (ESN) [2] or
liquid state machine (LSM) [3], has been reported to have great efficacy in
reconstruction or/and prediction of many complex physical systems only based
on the observational data of time series [4, 5, 6, 7]. The architecture of RC
is quite contracted. As shown in Fig. 1, only three weight matrices are
involved: the input matrix and the reservoir recurrent matrix are randomly
generated but fixed, while the output matrix is determined via training. As
such, efficient least squares optimization methods rather than the resource-
consuming back propagation algorithm are adopted in the training process [8].
Behind such a contracted architecture, two questions arise naturally: “What is
the fundamental mechanism resulting the efficacy of RC?” and “How to improve
the structure using the uncovered mechanism?” These questions have attracted
great attention and motivated abundant discussions, including those from the
topology and the complexity of random connections [9, 10] to the spectral
radius of random networks and the edge of chaos [11, 12, 13], from the fading
memory property [14] to the echo state property [15, 16], from the choice of
activation functions [17] to the training algorithm of the output layer [18].
Yet, recent understanding of RC is often via heuristic interpretation and it
is widely believed that a successful RC should possess high dimensionality,
nonlinearity, fading memory, and separation property [6], but barely with
rigorous and mathematical demonstrations.

In order to decipher the RC’s capacity of reconstructing and forecasting
nonlinear dynamics, several efforts from a viewpoint of dynamical systems have
been recently made. For example, the regression model and the dynamical model
decomposition method were used to illustrate the usefulness of RC to
forecasting chaotic dynamics [19, 20], and, to demonstrate the approximation
capability of RC, an embedding conjecture was studied and could be partially
validated for a specific form of RC under right technical conditions [21, 22].
In the area of photonic neural network, an architecture of photonic reservoir
computing has been developed through using a spatiotemporal analogy to
translate a delayed differential equation (DDE) into a virtual single-neuron
reservoir network [23, 24, 25]. Still, despite these significant efforts and
achievements, some key questions remain unsolved: “How to understand the
network dimension of general RC using the theories of nonlinear dynamics and
functional analytics?” and “How to design a small size network in RC for
sustaining its efficacy?”

Figure 1:  RCs as different nonlinear dynamical systems and embeddings. (a) A
standard RC without time-delay. (b) The generalized embedding $\Psi$ from the
input dynamics to the standard non-delayed reservoir network and the delayed
embedding $F$ from the input dynamics to the delayed reservoir network, which
constitute a topological conjugation between the dynamics of the non-delayed
reservoir network and the delayed reservoir network. (c) A time-delayed RC
with a smaller network size in the reservoir layer.

In this Letter, we rigorously study the mechanism of RC from a viewpoint of
nonlinear dynamical systems and novelly propose a framework of time-delayed
RC. Particularly combining the delayed embedding theory with the generalized
embedding theory, we first prove a general reservoir network rigorously as a
high dimensional embedding of the original input nonlinear dynamical system.
Then, we further reveal a trade-off relation between the time delays and the
number of neurons by unifying into a universal framework the standard RC
without delays and the time-delayed RC where the time delays are introduced
into the network’s output layer. It therefore allows us to construct a random
reservoir network with a significantly-reduced physical dimension to achieve
the efficacy that the original larger-size RC owns. Surprisingly, we show that
a standard reservoir of single-neuron, without introducing any DDE or time-
division multiplexing technique, can sometimes work well for reconstructing
and forecasting some representative physical systems. Moreover, we find
flexible memory capacity in the time-delayed RC, which makes it possible to
accomplish more challenging tasks of dynamics reconstruction that cannot be
easily achieved using a standard RC of the same scale.

We start with a standard RC as sketched in Fig. 1. Here, the input data
$\bm{x}_{k}\in\mathbb{R}^{n}$ represents the state vector of a dynamical
system that is evolving on a compact manifold $\mathcal{M}$ with the evolution
operator $\varphi\in{\rm
Diff}^{2}(\mathcal{M}):\bm{x}_{k+1}=\varphi(\bm{x}_{k})$. The vector
$\bm{r}_{k}\in\mathbb{R}^{m}$ represents the state of $m$ reservoir neurons at
time step $k$, the input layer weight matrix $W_{{\rm in}}$ and the reservoir
network matrix $W_{{\rm res}}$ are, respectively, $m\times n$ and $m\times m$
random matrices generated according to certain distribution laws. The
dynamical evolution of the reservoir neurons is governed by (RN):
$\bm{r}_{k}=(1-\alpha)\bm{r}_{k-1}+\alpha\phi(W_{{\rm
res}}\bm{r}_{k-1}+W_{{\rm in}}\bm{x}_{k}),$ where $\alpha$ is the leakage
factor, and $\phi\in C^{2}(\mathbb{R},(-1,1))$ is set a sigmoid function
(e.g., $\tanh$) in this Letter. The output vector
$\bm{y}_{k}\in\mathbb{R}^{l}$ is determined by the output weight matrix
$W_{\rm out}\in\mathbb{R}^{l\times m}$ such that $\bm{y}_{k}=W_{\rm
out}\bm{r}_{k}$. In the task of nonlinear system reconstruction, given the
time series, denoted by $\bm{x}_{k},k=1,\cdots,N+1$, as training data, the
target is to train the output weight matrix $W_{\rm out}$ so as to approximate
the one-step dynamics prediction, i.e., $\bm{y}_{k}\approx\bm{x}_{k+1}$. To
achieve this, the output weight matrix $W_{\rm out}$ is generally calculated
by minimizing the loss function
$\mathcal{L}=\sum_{k=1}^{N}\|\bm{x}_{k+1}-W_{\rm
out}\bm{r}_{k}\|^{2}+\beta\|W_{\rm out}\|^{2}$ over the training data set,
where $\beta>0$, the $L_{2}$-regularization coefficient, is introduced to make
optimization robust. After training, one can fix the output weight matrix
$W_{\rm out}$ and redirect the output $\bm{y}_{k}=W_{\rm out}\bm{r}_{k}$ as an
approximation of $\bm{x}_{k+1}$ into the input layer of the network and thus
generate the autonomous dynamics for $\bm{x}_{k}$ with $k>N$.

To rigorously establish an embedding theory for RC, we consider directly the
evolution (RN) of the reservoir neurons with the leakage factor $\alpha=1$ as:

$\begin{array}[]{l}\bm{r}_{k+1,\bm{x}_{0}}^{\bm{b}_{0}}=\phi(W_{\textrm{res}}\bm{r}_{k,\bm{x}_{0}}^{\bm{b}_{0}}+W_{\textrm{in}}\varphi^{k+1}\bm{x}_{0}),~{}~{}k=0,1,\cdots,\end{array}$

and define a map as
$\mathfrak{G}^{k}[\bm{r}_{0},W_{\textrm{res}},W_{\textrm{in}}](\bm{x}_{0})=\bm{r}_{k,\bm{x}_{0}}^{\bm{b}_{0}}$.
Here, $\bm{r}_{0,\bm{x}_{0}}^{\bm{b}_{0}}=\bm{b}_{0}$,
$\bm{b}_{0}\in\mathbb{I}^{m}$, and $\mathbb{I}=(-1,1)$. Thus, we rigorously
have the following result.

###### Theorem 1

Let $m\geqslant 2\textrm{\rm dim}(\mathcal{M})+1$ and
$[\bm{r}_{0},W_{\textrm{\rm res}},W_{\textrm{\rm
in}}]\in\mathbb{I}^{m}\times\mathbb{R}^{m\times m}\times\mathbb{R}^{m\times
n}$ with $\mbox{\rm dim}(\mathcal{M})$ as the box-counting dimension of the
manifold $\mathcal{M}$. Then, there exists a number $k^{*}>0$, such that
$\mathfrak{G}^{k}[\bm{r}_{0},W_{\textrm{\rm res}},W_{\textrm{\rm in}}]\in
C^{1}(\mathcal{M},\mathbb{R}^{m})$ is generically an embedding for all
$k>k^{*}$.

Here, the generic conclusion in Theorem 1 means that, for all
$[\bm{r}_{0},W_{\textrm{res}},W_{\textrm{in}}]\in\mathcal{S}$ where
$\mathcal{S}\subset\mathbb{I}^{m}\times\mathbb{R}^{m\times
m}\times\mathbb{R}^{m\times n}$ is an open and dense set,
$\mathfrak{G}^{k}[\bm{r}_{0},W_{\textrm{res}},W_{\textrm{in}}]$ is an
embedding for any sufficiently large $k$. The detailed and rigorous proof with
respect to the $C^{1}$-topology is provided in Supplemental Information (SI)
[26]. Moreover, the echo state property, a necessary condition for
constructing an RC, requires that, with the general configuration $\\{W_{\rm
in},W_{\rm res},\phi\\}$, the evolutions (RN) of the reservoir neurons,
starting from any different initial values $\bm{r}^{(1)}_{0}$ and
$\bm{r}^{(2)}_{0}$, converge to the same dynamics, i.e.,
$\lim_{k\rightarrow\infty}\|\bm{r}^{(1)}_{k}-\bm{r}^{(2)}_{k}\|=0$ [15].
Hence, by virtue of Theorem 1, regardless of the choice of the initial value
$\bm{r}_{0}$, the dynamics of reservoir neurons is determined by the input
dynamics, i.e., there exists a unique embedding $\Psi$ such that
$\bm{r}_{k}=\Psi(\bm{x}_{k})$ after a transient phase, while each component
$r_{ik}=\Psi_{i}(\bm{x}_{k})$ implies that the dynamics of each neuron is an
observable of the original dynamics.

Figure 2:  (a) Reconstructed dynamics of the Lorenz system by a non-delayed RC
including $200$ neurons, a delayed RC #1 including $40$ neurons with uniformly
$5$ lags for each neuron, and a delayed RC #2 including $40$ neurons with
random lags for each neuron. Here, the time unit is expressed in the Lyapunov
time, and the random lags are generated by a distribution centered at $5$ as
shown in the inset. (b) System reconstruction test for the Lorenz system with
different combinations of $N_{\rm neuron}$ and $N_{\rm lag}$, where the
training MSE in a log-scale and the contour curves are, respectively,
highlighted. Here, $\tau=5$ and the sampling stepsize is $\Delta t=0.01$. All
the other parameter settings are introduced in [26].

In the standard RC investigated-above, $m$, the number of reservoir neurons
and also known as the reservoir dimension, is often required to be huge [8,
6]. To design a different RC framework, significantly reducing $m$, we
introduce time delays into the output layer, as sketched in Fig. 1. While all
the configuration $\\{W_{\rm in},W_{\rm res},\phi\\}$ and the input data
$\bm{x}$ are set in the same manners, the reservoir network is assumed to
include $q~{}(<m)$ neurons only. Thus, a new reservoir vector before the
output layer is designated as
$\tilde{\bm{r}}_{k}=[r_{1,k},r_{1,k-\tau},\cdots,r_{1,k-d_{1}\tau+\tau},\cdots,r_{q,k},\cdots,r_{q,k-d_{q}\tau+\tau}]^{\top},$
and, correspondingly, the output matrix $W_{\rm out}$ is calculated by
minimizing the $L_{2}$ loss function

$\tilde{\mathcal{L}}=\sum_{k=1}^{N}\|\bm{x}_{k+1}-W_{\rm
out}\tilde{\bm{r}}_{k}\|^{2}+\beta\|W_{\rm out}\|^{2}$

with $W_{\rm out}\in\mathbb{R}^{l\times d}$ and $d=\sum_{i=1}^{q}d_{i}$. Here,
the new reservoir vector $\tilde{\bm{r}}_{k}$ is formed by the lagged dynamics
of each neuron, i.e., $q$ neurons with each neuron contributing $d_{i}$ lagged
dynamics $[r_{i,k},r_{i,k-\tau},\dots,r_{i,k-d_{i}\tau+\tau}]$ where $\tau$ is
a time delay. Assigning $d$ as the output dimension of this delayed RC.

Now, we are in a position to demonstrate that the time-delayed RC with the
above-assigned $d$ has the same representation and computation ability as the
standard RC involving $m$ neurons without time delay under the same parameter
settings, as long as $d\thickapprox m$. Actually, based on the delayed
embedding theory and its applications [27, 28, 29, 30], an approximate
combination of the lagged observable can also generically form an embedding,
i.e., for smooth observational functions $\Psi_{1},\cdots,\Psi_{q}$,
$F(\bm{x})=\big{[}\Psi_{1}(\bm{x}),\Psi_{1}(\varphi^{-1}(\bm{x})),\cdots,\Psi_{1}(\varphi^{-(d_{1}-1)}(\bm{x})),\cdots,\Psi_{q}(\bm{x}),\\\
\Psi_{q}(\varphi^{-1}(\bm{x})),\cdots,\Psi_{q}(\varphi^{d_{q}-1}(\bm{x}))\big{]}$
is generically an embedding as long as
$\sum_{i=1}^{q}d_{i}>2\textrm{dim}(\mathcal{M})$. Using the above-obtained
conclusion that each neuron is generally an observable, we further conclude
that the proposed new reservoir vector $\tilde{\bm{r}}_{k}$ is also an
embedding. Thus, the dynamics of the state vector ${\bm{r}}_{k}$ in the
$m$-neuron reservoir network without time delay is topologically conjugated
with the dynamics of the reservoir vector $\tilde{\bm{r}}_{k}$ of a $q$-neuron
reservoir network in the sense of embedding as long as $m=d$ with
$d=\sum_{i=1}^{q}d_{i}$, as sketched in Fig. 1. Consequently, we come to a
conclusion that the delayed observables of the RC state, seen as additional
nonlinear observables, have the same computational power in the system
reconstruction.

To demonstrate the capability of our time-delayed RC, we first consider the
benchmark Lorenz system. After a training phase including $N=6,000$ samples,
the autonomously-generated dynamics by the RC are shown in Fig. 2.
Particularly, used are a standard RC, a time-delayed RC containing fewer
neurons with uniformly lagged dynamics for each neuron, and a time-delayed RC
containing the same number of neurons but with random lags for each neuron.
Clearly, the time-delayed RC has almost the same performance of system
reconstruction as the non-delayed one, no matter the lags are uniformly or
randomly generated. Actually, this coincides with the above-performed
arguments from a viewpoint of embedding that the dynamics of this non-delayed
RC is a generalized embedding to the input dynamics with generically $200$
observables, while the dynamics of the time-delayed RC forms an embedding of
dimension $200$ when the sum of lags equals $200$ for either uniform or random
lags. Such a trade-off relation is further clearly illustrated in Fig. 2 where
different neuron number with different lag number for each neuron is combined,
and, for each combination, a training error is calculated as the mean squared
error (MSE) on the training data set based over $20$ independent runs. As
depicted in Fig. 2, for a fixed moderate number of neurons, the training error
decreases monotonically with the lag number for each neuron, and, for a fixed
moderate lag number, the training error also decreases monotonically with the
neuron number. Analogous results are also obtained for the other benchmark
systems, as presented in [26] (see Fig. S5). All these further reinforce the
above conclusion that, whenever $d\thickapprox m$ with moderate $N_{\rm lag}$
and $N_{\rm neuron}$, the time-delayed and the non-delayed RCs generally share
the same ability in system representation, and the numbers of neurons and of
time lags can be traded off mutually in these frameworks.

Such a trade-off relationship further puts the non-delayed and time-delayed
RCs into a unified framework where the output dimension $d$ becomes the
effective reservoir dimension that finally decides the ability of the system
representation. The standard non-delayed RC is actually a degenerated form in
this unified framework where all the neurons have zero lag. More surprisingly,
we find that it is even possible to reduce the number of neurons into one and
realize a single-neuron reservoir in the proposed framework. To see this, we
consider a gene regulation model with multiple delays:
$\dot{x}(t)=-kx(t)+gf_{1}(x(t-\tau_{1}))f_{2}(x(t-\tau_{2}))$ which describes
self-inhibitory and self-activation with distinct delays $\tau_{1}$ and
$\tau_{2}$, and with specific parameters the one-dimensional model has chaotic
dynamics [31, 32, 26]. Specifically, a time-delayed RC including only one
neuron with $600$ lags is used to reconstruct the dynamics, and the
autonomously generated dynamics after training are shown in Fig. 3. The
results confirm that the single-neuron, time-delayed RC performs well,
achieving the same reconstruction ability of the time-delayed RC with multiple
neurons. Frankly, the single-neuron RC in this numerical illustration is only
a special case that is not universally suitable for any system reconstruction.
Due to multi-scale property, the task of system reconstruction for multi-
variable using one reservoir network usually requires more than one single-
neuron. As for the task in Fig. 2, in order to get a successful prediction for
the $3$ components of the Lorenz system, a single-neuron reservoir is not
adequate even with multiple time delays.

Figure 3:  (a) Reconstructed dynamics of the chaotic gene regulation model by
the standard RC including 600 neurons (see the inset), the single-neuron,
time-delayed RC#1 with 600 lags, and the time-delayed RC#2 including 6 neurons
and 100 lags for each neuron. (b) MC test with different combinations of
$N_{\textrm{neuron}}\cdot N_{\textrm{lag}}$. Here, $\tau=5$ and the sampling
stepsize is $\Delta t=0.1$. All the other parameter settings are introduced in
[26].

In addition to the equivalent representation ability in the sense of
embedding, we further discover that the time-delayed RC has a more flexible
memory capacity which is an essential measure for RC’s reconstruction ability
for delayed systems. In the dynamics reconstruction job for the above gene
regulation model in Fig. 3, the chaotic dynamics cannot be reconstructed by a
standard RC, no matter how large the reservoir is, according to the dimension
test [26]. However, with all the same reservoir environment, the time-delayed
RC [both RC#1 and RC#2 in Fig. 3] can fulfill the job quite well. To
understand this phenomenon, we calculate the memory capacity (MC) for
different RC frameworks, using the definition in [8] and with different
combinations of $N_{\textrm{neuron}}$ and $N_{\textrm{lag}}$ but satisfying
the same output dimension, i.e., $N_{\textrm{neuron}}\cdot
N_{\textrm{lag}}=600$. Specifically, MC of a reservoir refers to its ability
to retain information from previous time steps and it is defined in [8] as

$\displaystyle{\rm
MC}_{k}=\frac{\operatorname{cov}\left(x(t-k),\hat{y}_{k}(t)\right)^{2}}{\operatorname{var}(x(t))\cdot\operatorname{var}\left(\hat{y}_{k}(t)\right)},$

where a random sequence of input values $x(t)$ is presented to the reservoir,
and the reservoir output $\hat{y}_{k}(t)$ is trained to predict a previous
input value $x(t-k)$, and here $\text{cov}(\cdot)$ and $\text{var}(\cdot)$,
respectively, represent covariance and variance.

Figure 3 clearly shows that, as $N_{\textrm{lag}}$ increases, the reservoir
computer with different delay settings has stronger memory capacity though
still keeping a fading memory fashion. This is essential for the dynamics
reconstruction job particularly for time-delayed physical or biological
systems such as the gene regulation model above. Thus, the proposed time-
delayed RC framework has a more flexible capability to deal with dynamics
reconstruction jobs requiring tunable MC.

Finally, to further validate the efficacy of the time-delayed RC in
reconstructing high-dimensional spatial-temporal system, we consider the ideal
storage cellular automation model (ISCAM) simulating heterocatalytic reaction-
diffusion processes at metal surfaces [33, 34]. Considering the extremely high
dimension (the $100\times 100$ grids yields $10000$ input dimension), it is a
challenging job to reconstruct the chaotic spatial-temporal patterns. As shown
in Fig. 4, with the same reservoir output dimension, the time-delayed RC has
almost the same reconstruction ability as the non-delay one.

Our framework uses a few hyper-parameters, such as $d$, the effective
reservoir dimension, and $\tau$, the time delay, which definitely affect RC’s
efficacy in system reconstruction. In fact, the existing literature included
some criteria for selecting such parameters in system reconstruction using
delayed embedding theory. We thus implement these criteria, the dimension test
and the delayed mutual information (DMI), to determine $d$ and $\tau$. From a
perspective of embedding, $d$ is only required to be larger than
$2\cdot\textrm{dim}(\mathcal{M})$ while practically the box-counting dimension
of the manifold $\mathcal{M}$ is usually very small, i.e.,
$\textrm{dim}(\mathcal{M})$ is between $2$ to $3$ [35, 36] for the chaotic
Lorenz attractor. However, to design an effective RC, $d$ is required to be
moderately large (see all the examples above). This is probably due that,
although the generic property in the embedding theory means open and dense in
a topological sense, there are still degenerated situations in practice,
particularly for randomly-generated networks (see Fig. S1 in [26]). Moreover,
to reveal the mechanism from representation to computation, the recent efforts
used the universal approximation theory [21] and the DMD [19] framework which
further demonstrate the necessity of a large network size of RC in achieving
good approximations. Thus, the dimension tests are used to seek a suitable $d$
for each computation. As for the delay $\tau$, either too small or too large
value renders computation problematic in system reconstruction, which
naturally prompts us to introduce a modified DMI test taking into account the
intrinsic time-scales of the neuronal dynamics in RC. Finally, it is noted
that, for chaotic systems, the lagged observables earlier than the Lyapunov
time have diminishing predictive power for the current time step, so we
suggest the constraint $\tau\cdot\Delta t\cdot N_{\rm lag}<\Lambda_{\max}$ for
the choice of $\tau$ and $N_{\rm lag}$ in practice where $\Delta t$ is the
sampling stepsize. The details for the choice of these hyper-parameters are
referred to [26].

Figure 4:  Reconstructed chaotic patterns for the ISCAM model by a standard
non-delayed RC including 5000 neurons and a time-delayed RC including 1000
neurons and 5 lags for each neuron. (a) Selected dynamical pattern using
different evolution rules. (b) Reconstruction errors deviating from true
dynamics from different RC frameworks.

In conclusion, we have provided a deep and rigorous insight to the mechanism
of RC from a viewpoint of embedding theory and nonlinear dynamical systems.
Based on our analytical findings, we have studied the role of time delay in
the reservoir network and proposed a new framework of time-delayed RC. This
framework can significantly reduce the network size and promote the memory
capacity, making its ability attain or even transcend the ability owned by the
standard RC. Considering the computational costs which are crucially dependent
on the network size in the dynamical evolution of RC and the hardware costs
related to the circuit size in those overwhelmingly-developed physical RCs
[6], smaller-size reservoir is always expected to promote its real and
extensive applications. Moreover, we notice a recently-published and
independent work [37], where a method, different from the perspective of
embedding theory and memory capacity presented here, was proposed to
concatenating internal states through time in RC and realize model-size
reduction. Lastly, any contributions to designing RC frameworks of low-
resource-consumption are believed to advance the direction of machine learning
and thus be of broad applicability in solving data-driven science and
engineering problems.

This work is supported by the National Natural Science Foundation of China
(Grant nos. 11925103 and 12171350), and by the STCSM (Grant nos. 18DZ1201000
and 2021SHZDZX0103).

## References

  * Jaeger and Haas [2004] H. Jaeger and H. Haas, Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science 304, 78 (2004).
  * Jaeger [2001] H. Jaeger, The “echo state” approach to analysing and training recurrent neural networks-with an erratum note, German National Research Center for Information Technology GMD Technical Report 34, 148 (2001).
  * Maass  _et al._ [2002] W. Maass, T. Natschläger, and H. Markram, Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Computation 14, 2531 (2002).
  * Pathak  _et al._ [2018] J. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott, Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach, Physical Review Letters 120, 024102 (2018).
  * Zhu  _et al._ [2019] Q. Zhu, H. Ma, and W. Lin, Detecting unstable periodic orbits based only on time series: When adaptive delayed feedback control meets reservoir computing, Chaos: An Interdisciplinary Journal of Nonlinear Science 29, 093125 (2019).
  * Tanaka  _et al._ [2019] G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose, Recent advances in physical reservoir computing: A review, Neural Networks 115, 100 (2019).
  * Tang  _et al._ [2020] Y. Tang, J. Kurths, W. Lin, E. Ott, and L. Kocarev, Introduction to focus issue: When machine learning meets complex systems: Networks, chaos, and nonlinear dynamics, Chaos: An Interdisciplinary Journal of Nonlinear Science 30, 063151 (2020).
  * Jaeger [2002] H. Jaeger,  _Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the” echo state network” approach_ , Vol. 5 (GMD-Forschungszentrum Informationstechnik Bonn, 2002).
  * Haluszczynski and Räth [2019] A. Haluszczynski and C. Räth, Good and bad predictions: Assessing and improving the replication of chaotic attractors by means of reservoir computing, Chaos: An Interdisciplinary Journal of Nonlinear Science 29, 103143 (2019).
  * Griffith  _et al._ [2019] A. Griffith, A. Pomerance, and D. J. Gauthier, Forecasting chaotic systems with very low connectivity reservoir computers, Chaos: An Interdisciplinary Journal of Nonlinear Science 29, 123108 (2019).
  * Lukoševičius and Jaeger [2009] M. Lukoševičius and H. Jaeger, Reservoir computing approaches to recurrent neural network training, Computer Science Review 3, 127 (2009).
  * Jiang and Lai [2019] J. Jiang and Y.-C. Lai, Model-free prediction of spatiotemporal dynamical systems with recurrent neural networks: Role of network spectral radius, Physical Review Research 1, 033056 (2019).
  * Boedecker  _et al._ [2012] J. Boedecker, O. Obst, J. T. Lizier, N. M. Mayer, and M. Asada, Information processing in echo state networks at the edge of chaos, Theory in Biosciences 131, 205 (2012).
  * Dambre  _et al._ [2012] J. Dambre, D. Verstraeten, B. Schrauwen, and S. Massar, Information processing capacity of dynamical systems, Scientific Reports 2, 1 (2012).
  * Yildiz  _et al._ [2012] I. B. Yildiz, H. Jaeger, and S. J. Kiebel, Re-visiting the echo state property, Neural networks 35, 1 (2012).
  * Manjunath and Jaeger [2013] G. Manjunath and H. Jaeger, Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks, Neural Computation 25, 671 (2013).
  * Chang  _et al._ [2019] H. Chang, S. Nakaoka, and H. Ando, Effect of shapes of activation functions on predictability in the echo state network, arXiv preprint arXiv:1905.09419 https://doi.org/10.48550/arXiv.1905.09419 (2019).
  * Steil [2007] J. J. Steil, Online reservoir adaptation by intrinsic plasticity for backpropagation–decorrelation and echo state learning, Neural Networks 20, 353 (2007).
  * Bollt [2021] E. Bollt, On explaining the surprising success of reservoir computing forecaster of chaos? the universal machine learning dynamical system with contrast to var and dmd, Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 013108 (2021).
  * Gauthier  _et al._ [2021] D. J. Gauthier, E. Bollt, A. Griffith, and W. A. S. Barbosa, Next generation reservoir computing, Nature Communications 12, 5564 (2021).
  * Hart  _et al._ [2020] A. Hart, J. Hook, and J. Dawes, Embedding and approximation theorems for echo state networks, Neural Networks 128, 234 (2020).
  * Hart  _et al._ [2021] A. Hart, J. Hook, and J. Dawes, Echo state networks trained by tikhonov least squares are l2 ($\mu$) approximators of ergodic dynamical systems, Physica D: Nonlinear Phenomena 421, 132882 (2021).
  * Appeltant  _et al._ [2011] L. Appeltant, M. C. Soriano, G. Van der Sande, J. Danckaert, S. Massar, J. Dambre, B. Schrauwen, C. R. Mirasso, and I. Fischer, Information processing using a single dynamical node as complex system, Nature Communications 2, 468 (2011).
  * Brunner  _et al._ [2018] D. Brunner, B. Penkovsky, B. A. Marquez, M. Jacquot, I. Fischer, and L. Larger, Tutorial: Photonic neural networks in delay systems, Journal of Applied Physics 124, 152004 (2018).
  * Larger  _et al._ [2017] L. Larger, A. Baylón-Fuentes, R. Martinenghi, V. S. Udaltsov, Y. K. Chembo, and M. Jacquot, High-speed photonic reservoir computing using a time-delay-based architecture: Million words per second classification, Physical Review X 7, 011015 (2017).
  * [26] see supplementary materials for details.
  * Takens [1981] F. Takens, Detecting strange attractors in turbulence, Lecture Notes in Mathematics 898, 366 (1981).
  * Sauer  _et al._ [1991] T. Sauer, J. Yorke, and M. Casdagli, Embedology, Journal of Statistical Physics 65, 579 (1991).
  * Ma  _et al._ [2018] H.-F. Ma, S. Leng, K. Aihara, W. Lin, and L. Chen, Randomly distributed embedding making short-term high-dimensional data predictable, Proc. Natl. Acad. Sci. USA 115, E9994 (2018).
  * Hou  _et al._ [2022] J.-W. Hou, H.-F. Ma, D. He, J. Sun, Q. Nie, and W. Lin, Harvesting random embedding for high-frequency change-point detection in temporal complex systems, National Science Review 9, nwab228 (2022).
  * Suzuki  _et al._ [2016] Y. Suzuki, M. Lu, E. Ben-Jacob, and J. N. Onuchic, Periodic, quasi-periodic and chaotic dynamics in simple gene elements with time delays, Scientific Reports 6, 1 (2016).
  * Wernecke  _et al._ [2019] H. Wernecke, B. Sándor, and C. Gros, Chaos in time delay systems, an educational review, Physics Reports 824, 1 (2019).
  * Dress  _et al._ [2010] A. W. Dress, W. Hordijk, W. Lin, and P. Serocka, The ideal storage cellular automaton model, in  _Dagstuhl Seminar Proceedings_ (Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2010).
  * Dress and Lin [2011] A. W. Dress and W. Lin, Dynamics of a discrete-time model of an ”ideal-storage” system describing hetero-catalytic processes on metal surfaces, International Journal of Bifurcation and Chaos 21, 1331 (2011).
  * McGuinness [1983] M. J. McGuinness, The fractal dimension of the lorenz attractor, Physics Letters A 99, 5 (1983).
  * Viswanath [2004] D. Viswanath, The fractal property of the lorenz attractor, Physica D: Nonlinear Phenomena 190, 115 (2004).
  * Sakemi  _et al._ [2020] Y. Sakemi, K. Morino, T. Leleu, and K. Aihara, Model-size reduction for reservoir computing by concatenating internal states through time, Scientific Reports 10, 1 (2020).