# Event-based Camera Tracker by $\nabla_{t}$NeRF

Mana Masuda∗, Yusuke Sekikawa†, Hideo Saito∗
∗Keio University, †Denso IT Laboratory
{mana.smile<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

When a camera travels across a 3D world, only a fraction of pixel value
changes; an event-based camera observes the change as sparse events. How can
we utilize sparse events for efficient recovery of the camera pose? We show
that we can recover the camera pose by minimizing the error between sparse
events and the temporal gradient of the scene represented as a neural radiance
field (NeRF). To enable the computation of the temporal gradient of the scene,
we augment NeRF’s camera pose as a time function. When the input pose to the
NeRF coincides with the actual pose, the output of the temporal gradient of
NeRF equals the observed intensity changes on the event’s points. Using this
principle, we propose an event-based camera pose tracking framework called
TeGRA which realizes the pose update by using the sparse event’s observation.
To the best of our knowledge, this is the first camera pose estimation
algorithm using the scene’s implicit representation and the sparse intensity
change from events.

##  1 Introduction

Camera localization/tracking is one of the fundamental functionality of
computer vision. The field of use lies in many applications like automotive,
augmented reality, and robotics. Event-based cameras detect sparse intensity
changes with extremely high temporal resolution (>$10,000$ fps). This unique
feature makes it a suitable sensor for tracking fast-moving scenes, and many
researchers have been exploring several approaches to utilize high-speed
observation. Recently, [21, 10] showed the pose & motion is recovered by
minimizing the error between the estimated intensity change and integrated
events (Fig.1). Thanks to the low-latency nature of events, their method works
well even in a rapid camera motion. However, they need a dense operation to
differentiate the error w.r.t pose & motion; it could not take advantage of
the sparsity of events. Therefore, the computational cost increases linearly
with the processing rate, making it difficult to run the algorithm in real-
time on devices with limited computational resources.

Figure 1: $\operatorname{argmin}$ $|$Temp. Grad. of  NeRF($S_{t}$) $-$
Event$|$ is  Pose & Motion.

NeRF [37] is an implicit light-field representation using a neural network,
which enables unified perceptions of the 3D world that is difficult for the
existing explicit MAP (e.g., CAD model). Many NeRF-extention have been
proposed [54] to model a complex 3D scene; we believe these advancements make
the NeRF representation a novel candidate for representing 3D MAP for camera
localization/tracking. Utilization of NeRF as a 3D MAP for camera pose
localization has already been explored [62, 30]. Their basic idea for
realizing camera pose estimation is minimizing the error between the estimated
intensity frame from NeRF and the observed intensity frame, w.r.t input camera
pose (Fig.1). However, these methods can not estimate the camera pose by using
sparse intensity change.

These existing studies motivate one question; how can we utilize sparse events
to recover the camera pose without converting them into dense frames? We show:

Pose is recovered by minimizing $|$$\nabla_{t}$NeRF($S_{t}$) - event$|$.

When coordinates of events (where the event camera detects intensity changes)
and the current camera pose estimate on the event’s time $S_{t}$ are input to
NeRF, it outputs the intensities at that point and time. By viewing the input
camera pose to NeRF as a function of time, we found that the temporal gradient
of the intensity w.r.t the event’s timestamp is the estimation of intensity
changes on that pixel. Pose & motion is obtained by minimizing $|$estimated
intensity changes $-$ observed intensity changes (events)$|$ (Theorem-1,
Fig.1).

Based on this principle, we propose an event-based camera pose tracking
framework called minimization of the $\nabla_{t}$NeRF($S_{t}$) (TeGRA). Unlike
the conventional dense approach (frame-based), which operates in intensity-
space, TeGRA works sparsely (event-based) in gradient-space; it updates the
pose & motion by evaluating the error only for the pixels where the events
have been observed. By the sparse mechanism, the number of pixels to be
evaluated is 99.8% less than the conventional dense algorithm on our created
event dataset. To the best of our knowledge, this is the first approach to
realizing camera pose estimation using the implicit representation of the 3D
scene and the sparse observation of intensity changes. We provide the
theoretical proof of TeGRA. Furthermore, we created a photo-realistic event
dataset for 6DoF camera pose tracking with a ground-truth pose called EvTrack
(EVent-based TRACKing dataset). Using the EvTrack, we experimentally proved
the concept.

Tracking using event stream and explicit scene  Tracking using intensity frame
and implicit scene  Tracking using event stream and implicit scene (TeGRA)

##  2 Related Work

###  2.1 Camera Pose Tracking from Intensity Frame

There is a long history in the field of camera pose tracking. Many algorithms
are based on frame-based observation, namely video sequence. Methods based on
KLT [34, 5] are among the most popular visual tracking algorithms. KLT
computes the camera pose by aligning the observed intensity frame with the
known scene’s intensity map using an image gradient. Recently, the deep neural
network (DNN) based feature extractor has been utilized to exploit more rich
features [13] than the raw pixel value. This KLT-based method works well in
many scenarios. However, it easily collapses in a fast-moving scene, which
induces a significant difference between the observed frame and the estimated
frame. The difference makes the gradient-based algorithms trapped into a local
minimum. Another line of research utilizes DNN to directly regress the pose
between pairs of images [7, 26, 8, 28, 55]. They have an advantage in
computational efficiency (over the iterative gradient-based algorithm) because
they can update the pose by a single forward pass of the network. They may
also suffer from performance degeneration when there is a significant
difference in the pairs of images due to the fast camera motion.

Either gradient-based or regression-based, the problem due to the fast camera
motion might be mitigated by using an expensive high-speed camera. However,
processing them at a higher rate is infeasible due to increased computational
complexity.

###  2.2 Camera Pose Tracking from Events

An event-based camera [14, 43, 44, 53, 49] is a bio-inspired vision sensor
that mimics biological retinas. It differs from a frame-based camera in its
H/W design; its report per-pixel intensity changes as asynchronous event
streams. Thanks to this unique operation principle, event-based cameras have
significant advantages over conventional frame-based cameras, e.g., low
latency, high dynamic range (HDR), and blur-free. The most important feature
of the event-based camera for camera pose tracking would be its high temporal
resolution. Their temporal resolution is equivalent to >$10,000$ fps, making
it a suitable sensor for robust tracking in fast-moving scenes.

Use of Event frame for Camera Pose Estimation. There have been many attempts
to utilize the distinct features of the event-based camera for tracking [47,
67, 19, 21, 10, 1, 2, 12, 22]. Recently, KLT has been extended to event data
to realize robust camera pose tracking in high-speed and HDR scenarios [10,
21, 22]. They update camera pose and motion by minimizing the error in the
integrated event frame between estimation and observation (Fig.1). The
estimation is computed from the current estimate of the pose & motion using
the pre-build 3D intensity map. By utilizing the low-latency nature of events,
their methods work well even in rapid camera motion. However, computing the
derivatives of the error w.r.t pose requires computationally intensive dense
rendering of the 3D map. The dense computation needs to be repeated until
convergence for each frame. Therefore, running the algorithm in real-time on
devices with limited computational resources is difficult. Our goal in this
study is to realize an efficient sparse algorithm for camera pose tracking
using the implicit representation of a 3D scene.

###  2.3 Implicit Scene Representations

Representing data, such as image, video, and shape using implicit neural
representation gained much attention such as for data compression [15, 35],
novel-view synthesis [36, 37, 38], 3D-shape modeling [40, 11, 3, 50], and
image registration [62, 30] to name a few. NeRF [37] utilizes the implicit
neural representations to represent the 4D light-field. Given a set of images
paired with a camera pose, NeRF learns the intensities of each pixel for a
given camera pose. Each pixel’s intensity is computed by integrating the RGB
color and density along the corresponding ray using the volumetric rendering
technique. Due to its flexibility, NeRF extensions are exploding beyond its
original application of novel-view synthesis; e.g., 3D-shape reconstruction
[39, 56], disentangle lighting [51, 9, 59, 64], image editing [33, 25, 61],
object separation [60, 4], semantic label propagation [65], modeling time-
varying objects [45, 41, 18, 27, 32, 29], and depth estimation [24, 58].

Use of NeRF for Camera Pose Estimation These novel functionalities realized by
NeRF such as modeling lighting and moving objects would realize a unifying
perception of the 3D world that is difficult for the existing explicit MAP
(e.g., CAD model). Furthermore, recent progress of NeRF-extention enabled it
to model a large environment, such as a large house [17], and an entire city
[54]. We believe these advancements make the NeRF representation a novel
candidate for representing 3D MAP for camera localization/tracking. iNeRF [62]
uses the inverse of NeRF to estimate the camera pose. BARF [30], NeRF- [57],
iMAP [52], NICE-SLAM[66] realized simultaneous camera pose estimation and 3D
scene reconstruction. Their basic idea for realizing camera pose estimation is
minimizing the error between the estimated intensity frame from NeRF and the
observed intensity frame, w.r.t input camera pose (Fig.1). These methods can
not estimate the camera pose by using sparse intensity change. We aim to
derive a camera pose tracking algorithm using sparse intensity changes
observation (i.e., events) to realize efficient camera pose tracking.

##  3 Method

###  3.1 Preliminaries

Problem Statement Our goal in this study is to develop an efficient camera
pose tracking algorithm TeGRA using sparse observation of intensity-change
event (IC-event) stream $\mathbf{e}_{t}$ as follows:

$\mathrm{TeGRA}:(\mathbf{e}_{t},S^{\mathrm{ini}}_{t},\dot{S}^{\mathrm{ini}}_{t})\mapsto(S^{\mathrm{opt}}_{t},\dot{S}^{\mathrm{opt}}_{t}),$
(1)

where $(S^{\mathrm{ini}}_{t},\dot{S}^{\mathrm{ini}}_{t})$ and
$(S^{\mathrm{opt}}_{t},\dot{S}^{\mathrm{opt}}_{t})$ are the initial and
optimized pose & motion at time $t$ respectively. TeGRA utilizes a
differentiable implicit representation of the static 3D world.

NeRF This study assumes that the 3D scene is implicitly represented by NeRF
[37]. NeRF, $G_{\mathcal{M}}$ was initially proposed for novel-view synthesis;
it represents the scene using a neural network that is differentiable w.r.t to
its input pose ${S}$. The parameter of the NeRF, $\mathcal{M}$, is pre-trained
before tracking. NeRF $G_{\mathcal{M}}$ takes an image coordinate $(x,y)$ and
6DoF pose $S\in\mathfrak{s}\mathfrak{e}(3)$ of the camera as inputs and
renders the RGB intensity $\mathbf{c}$ at that coordinate;

$G_{\mathcal{M}}(x,y,S)=\mathbf{c}.$ (2)

IC-Event The IC-event stream $\mathbf{e}_{t}$ is a set of events observed in
time interval $[t,t+\tau]$ as follows:

$\mathbf{e}_{t}=[e^{1},...,e^{i},...,e^{M}];\ \
e^{i}=[e_{x}^{i},e_{y}^{i},e_{u}^{i},e_{r}^{i}]^{\mathrm{T}},$ (3)

where $(e_{x}^{i},e_{y}^{i})\in\mathbb{R}^{2}$ is the image coordinates where
the intensity change has been detected, $e_{u}^{i}\in\mathbb{R}$ is the
timestamp of the change, and $e_{r}^{i}\in\mathbb{R}$ is the intensity change
value. The intensity change value $e_{r}(x,y)$ on $(x,y)$ within time interval
$\Delta t$ is defined using true intensity $L(x,y,t)$ as follows:

$e_{r}(x,y)=\frac{L(x,y,t+\Delta t)-L(x,y,t)}{\Delta t}.$ (4)

Events are detected where the intensity change $e_{r}(x,y)$ exceeds the
predefined threshold $\delta$. For ease of discussion, we define
$\bar{e}_{u}^{i}$, which is relative timestamp w.r.t time $t$;
$\bar{e}_{u}^{i}:=e_{u}^{i}-t$. That is $\bar{e}_{u}^{i}$ changes depending on
the time $t$ we consider. Some event-based cameras directly detect the
intensity change $e_{r}$, such as Celex-V [14], or IC-event is obtained from
high-speed video data. We left the evaluation using the binary event for
future work111At this time, we consider there are two possible approaches to
make TeGRA compatible with the binary-event; 1) Modify the loss function of
(5) to adapt to the binary-event (Sec.5.2), 2) Convert the binary-event to IC-
event using the timestamp (supplement-D). In this work, we use IC-event for
simplicity and left the exploration for future work. .

Figure 2: Pose & motion estimation (visual explanation of theorem-1)

###  3.2 The answer of $\nabla_{t}$NeRF($S_{t}$)= 0 is true pose & motion

By viewing the input camera pose to NeRF as a function of time, we found that
camera pose & motion $(S,\dot{S})$ can be recovered by minimizing the error
between the temporal gradient of the 3D-scene represented as NeRF
$G_{\mathcal{M}}$ and IC-event stream $\mathbf{e}$ (Fig.2: visual
explanation).

######  Theorem 1.

The minimizer of $|\nabla_{t}\mathrm{NeRF}(S_{t})-\mathrm{event}|$ is true
pose $\&$ motion ($S^{\mathrm{gt}}_{t},\dot{S}^{\mathrm{gt}}_{t}$):

$\begin{split}&S^{\mathrm{gt}}_{t},\dot{S}^{\mathrm{gt}}_{t}\\\
&=\mathop{\mathrm{argmin}}\limits_{S,\dot{S}}\underbrace{\sum_{i}\left\|\frac{\partial
G_{\mathcal{M}}\left(e_{x}^{i},e_{y}^{i},S_{t}+\bar{e}_{u}^{i}\dot{S}_{t}\right)}{\partial\bar{e}_{u}^{i}}-e_{r}^{i}\right\|_{2}}_{:=\mathcal{L}}.\end{split}$
(5)

###### Proof.

To prove the theorem-1, we’ll show the $i$-th element of the Eq.(5) equals to
zero when we have the true pose & motion
$({S}_{t}^{\mathrm{gt}},{\dot{S}}_{t}^{\mathrm{gt}})$. Now, consider the
temporal gradient of the NeRF of Eq.(2) for $i$-th event on time $e_{u}^{i}$:

$\begin{split}&\frac{\partial
G(e_{x}^{i},e_{y}^{i},{S}_{t}+\bar{e}_{u}^{i}\dot{{S}}_{t})}{\partial\bar{e}_{u}^{i}}\\\
&=\lim_{\Delta u\rightarrow
0}\frac{G(e_{x}^{i},e_{y}^{i},S_{t}+(\bar{e}_{u}^{i}+\Delta
u)\dot{S}_{t})-G(e_{x}^{i},e_{y}^{i},S_{t}+\bar{e}_{u}^{i}\dot{S}_{t})}{\Delta
u}\end{split}$ (6)

From the defenition of NeRF of Eq.(2), $L(x,y,t)$ equals to the NeRF’s output,
when the estimated pose & motion is true:

$L(x,y,t+\bar{e}_{u}^{i})=G(x,y,S_{t}^{\mathrm{gt}}+\bar{e}_{u}^{i}\dot{{S}}_{t}^{\mathrm{gt}}).$
(7)

Pluging this relation to Eq.(6)

$\begin{split}&\frac{\partial
G(e_{x}^{i},e_{y}^{i},S_{t}^{\mathrm{gt}}+\bar{e}^{i}\dot{S}_{t}^{\mathrm{gt}})}{\partial
e_{u}^{i}}\\\ &=\lim_{\Delta u\rightarrow
0}\frac{L(e_{x}^{i},e_{y}^{i},t+\bar{e}_{u}^{i}+\Delta
u)-L(e_{x}^{i},e_{y}^{i},t+\bar{e}^{i})}{\Delta u}\\\ &=\lim_{\Delta
u\rightarrow 0}\frac{L(e_{x}^{i},e_{y}^{i},e_{u}^{i}+\Delta
u)-L(e_{x}^{i},e_{y}^{i},e_{u}^{i})}{\Delta u}\\\ \end{split}$ (8)

This equals to the definition of $e_{r}^{i}$ of Eq.(4) when $\Delta t$ is
sufficiently small. ∎

###  3.3 Sparse Tracking Algorithm: TeGRA

Using the theorem-1, we propose an event-based camera pose-tracking algorithm
called TeGRA. The algorithm minimizes $\mathcal{L}$ in Eq.(5) using gradient-
decent (Fig. 1). The input to TeGRA is the IC-event stream $\mathbf{e}_{t}$ at
time $t$ and the initial estimate of pose & motion
($S^{ini}_{t},\dot{S}^{ini}_{t}$). To obtain the derivative to update pose &
motion, we differentiate the loss $\mathcal{L}$ w.r.t $(S_{t},\dot{S}_{t})$
through the temporal gradient of $G_{\mathcal{M}}$ on each event’s timestamp
$e_{u}^{i}$. Then, pose & motion is updated using the sum of all events
contributions. See listing LABEL:code for pseudo-PyTorch code.

⬇

1# event: Observed IC-event stream, pose, motion: Initial estimates

2def TeGRA(event, pose, motion, n_itr=100, eta=5e-4):

3 for itr in range(max_itr):

4 # Compute intensity at event pixel

5 rgb_est = G(event.x, event.y, pose + event.u * motion)

6 # Compute gradient of ‘rgb_est’ w.r.t event timestamp

7 gradt_est = torch.autograd.grad(rgb_est, event.u)

8 # Compute MSE between estimation and observation

9 loss = mse_loss(gradt_est, event.r)

10 # Compute derivative w.r.t pose and motion

11 loss.backward()

12 # Update pose and motion

13 pose += eta*pose.grad; motion += eta*motion.grad

14 return pose, motion

Listing 1: Pseudo-PyTorch Implementation of TeGRA

##  4 Experiments

To demonstrate the effectiveness of the proposed TeGRA, we created the 6DOF
camera pose tracking dataset for event data, called EvTrack, with ground-truth
camera pose. We used BOP challenge 2020 scenes [23] because it is photo-
realistic, and the scene well represents the indoor localization scenario. The
BOP uses the BlenderProc [16] to render realistic images using ray tracing. We
first show the tracking result of TeGRA for proof of concept (POC) (Sec.4.3).
Next, we show the intensive qualitative comparison with a dense algorithm
(Fig.1) using intensity in terms of pose estimation accuracy (Sec.4.4).

###  4.1 Event-based camera pose tracking dataset (EvTrack)

The dataset consists of five scenes, mix, hb, lm, tyol and ycbb, the last four
scenes correspond to the BOP data split and mix scenes include all the four
data to simulate the ordinary indoor situation. We used mix scene for proof of
concept in the tracking scenario and use the other four for the quantitative
evaluation in terms of pose estimation accuracy. We generated 500 images for
NeRF training and three camera trajectories (Fig.3 right) for each of the five
scenes simulating the drone hovering around a room. The IC-event stream is
generated by using pairs of consecutive images (total of 1,000),
$\\{L(t),L(t+\nu)\\}$ ($L(t)\in\mathbb{R}^{H\times W}$), by subtracting them.
The size of image $(H,W)$ is $(480\times 640)$. The threshold $\delta$ for
triggering the event is set to $0.05$ (intensities are normalized to $[0,1]$)
for all scenes. In our experiment, we converted the RGB IC event to a
grayscale IC event since most event cameras detect grayscale intensity
changes. Each event stream is generated using five consecutive frames,
assuming the motion is approximately linear within the interval.

Figure 3: Camera pose tracking results from mix scene (Sec.4.3); ground-truth
(cyan) and optimized trajectory (magenta) (visualized in every 7 timestep).
Table 1: Pose estimation accuracy comparison of existing dense approach
(Fig.1) and our sparse approach (Fig.1, TeGRA).

|  | hb | lm | tyol | ycbv
---|---|---|---|---|---
|  | Rot. | Trans. | Rot. | Trans. | Rot. | Trans. | Rot. | Trans.
Dense | seq0 | 0.034 | 0.0002 | 0.028 | 0.0004 | 0.034 | 0.0003 | 0.028 | 0.0003
seq1 | 0.036 | 0.0002 | 0.562 | 0.0003 | 0.033 | 0.0002 | 0.567 | 0.0004
seq2 | 0.030 | 0.0002 | 0.273 | 0.0004 | 0.028 | 0.0002 | 0.269 | 0.0004
Sparse (ours) | seq0 | 0.217 | 0.0018 | 0.321 | 0.0047 | 0.350 | 0.0030 | 0.760 | 0.0021
seq1 | 0.448 | 0.0065 | 0.790 | 0.0102 | 0.477 | 0.0053 | 0.448 | 0.0072
seq2 | 0.338 | 0.0054 | 0.362 | 0.0088 | 0.425 | 0.0058 | 0.290 | 0.0051

###  4.2 Implementation

We use Pytorch [42] to train the NeRF-model and run TeGRA for tracking222See
supplement for network architecture of NeRF (supplement-A.2), and frame-based
dense tracking algorithm which we compare the accuracy (supplement-C)..

Training We follow the network settings from the original NeRF with minor
modifications; use the softplus activation for the volume density $\sigma$ as
recommended in BARF [30] for improved stability.

Tracking We add the RGB-to-gray layer (Sec.3.3) to use grayscale IC-events. We
randomly select 750 event pixels from the observed event and update the pose &
motion for n_itr (1,000) times; it amounts to 0.2% of the dense algorithm. The
learning rate $\eta$ of the $S,\dot{S}$ is set to $5\times 10^{-5}$
exponentially decaying to $5\times 10^{-6}$ toward n_itr. The temporal
gradient of intensity w.r.t the event timestamp $e_{u}$ is computed by using
Pytorch’s autograd.grad.

###  4.3 Proof of Concept (Tracking)

For the POC of the proposed idea, we applied TeGRA for tracking. We used seq0
from mix scene. We use estimated pose & motion from the previous timestep to
initialize $(S^{\mathrm{ini}}_{t},\dot{S}^{\mathrm{ini}}_{t})$ in the next
timestep as discussed in Sec.3.3. The results are shown in Fig.3. We confirmed
that the pose is successfully tracked without drifting333Video for this
tracking is included in the supplement. See supplement-F for additional
results on NeRF dataset.. Finally, the pose estimation error was
$(0.13\degree,0.0003)$. The average number of events per pose update was 0.2%
of the entire pixel.

###  4.4 Quantitave Benchmarks (Pose Estimation)

To quantitatively evaluate the performance of the TeGRA, we compare the pose
estimation accuracy with the dense algorithm using the difference in intensity
(Fig.1). In this experiment, we randomly initialized the pose and motion for
each stream. The results are shown in Tab.1. Both achieved comparable accuracy
in maintaining camera-pose tracking, while ours used only 2.4% of pixels.

##  5 Conclusion

How can we utilize sparse events for recovering the camera pose? Answer:
Camera pose is recovered by minimizing the error between the temporal gradient
of the scene represented as a NeRF and sparse events. Our tracking algorithm,
TeGRA, could update the pose using sparse event points. This mechanism is a
significant advantage over the existing image-space algorithm, which requires
dense computation. We demonstrate TeGRA in a tracking scenario with unseen
background clutter.

We believe the proposed idea opens the door for realizing an event-based
camera pose tracking using implicit 3D-scene representation. This study
focuses on demonstrating the algorithm in a naive setup; therefore, we left
large areas for future work, either from an experimental or algorithmic
perspective.

###  5.1 Application to Real-World Data

One of our ultimate goals is to utilize the proposed method in the practical
autonomous driving scenario. More specifically, as future work, we plan to
apply the TeGRA for Block-NeRF [54], which realized scaling up the NeRF into a
city-scale automotive environment. The TeGRA could incorporate the mip-NeRF
[6] rendering algorithm, which is key to realizing the large-scale modeling in
Block-NeRF. The principle we present (theorem-1) is compatible with a variety
of NeRF-variant as long as the 3D-scene is represented in the form of Eq.(2),
where $G_{\mathcal{M}}$ is differentiable w.r.t pose $S$.

###  5.2 Asynchronous Update using Binary-Event

In this study, we use IC-event instead of binary-event, and synchronous event
stream instead of asynchronous one. We choose this experimental setup mainly
due to the implementational difficulties in generating event streams. It
requires engineering effort to generate asynchronous binary-event streams,
such as modifying event-camera simulators like ESIM [46]. To make the
algorithm compatible with the binary polarity $e_{p}$, the loss term in Eq.(5)
needs to be modified slightly:

$\mathcal{L}_{bin}={\sum_{i}\left\|\operatorname{SoftSgn}\left(\frac{\partial
G_{\mathcal{M}}\left(e_{x}^{i},e_{y}^{i},S_{t}+\bar{e}_{u}^{i}\dot{S}_{t}\right)}{\partial\bar{e}_{u}^{i}}\right)-e_{p}^{i}\right\|_{2}},$
(9)

where the $\operatorname{SoftSgn}$ is a soft version of the sign function,
which maps continuous intensity change into (soft) polarity. We left this
exploration as future work.

###  5.3 Speed Up

Thanks to the sparse mechanism of TeGRA, the number of pixels to be evaluated
for computing the pose update is significantly lower than the entire pixel
(Sec.4.3). Speeding up the NeRF for real-time rendering is an active research
topic [20, 31, 63, 48]. For example, FastNeRF [20] utilizes a separate network
for position-dependent MLP and direction-dependent MLP for speed up. As
discussed above, the proposed mechanism is compatible with other NeRF-
variants. We expect combining our sparse mechanism with these approaches is a
vital topic to realize real-time tracking on mobile devices. We’ll incorporate
the method and then evaluate the FLOPS and wall-clock time.

###  5.4 Extention to SLAM

It is an exciting research direction to extend our algorithm into simultaneous
localization and mapping (SLAM). Now, NeRF is emerging as an entirely new
framework for SLAM [52, 30, 57, 66]. iMAP [52] is pioneering work utilizing
NeRF for realizing real-time SLAM. We expect incorporating TeGRA will
significantly speed up the NeRF-based SLAM.

## References

  * [1] I. Alzugaray and M. Chli.  Asynchronous multi-hypothesis tracking of features with event cameras.  In Proc. Int. Conf. 3D Vis (DV), pages 269–278, 2019.
  * [2] Ignacio Alzugaray Lopez and Margarita Chli.  Haste: multi-hypothesis asynchronous speeded-up tracking of events.  In Proc. Brit. Mach. Vis. Conf. (BMVC), page 744, 2020.
  * [3] Matan Atzmon and Yaron Lipman.  Sal: Sign agnostic learning of shapes from raw data.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 2565–2574, 2020.
  * [4] Hendrik Baatz, Jonathan Granskog, Marios Papas, Fabrice Rousselle, and Jan Novák.  Nerf-tex: Neural reflectance field textures.  In Eurographics Symposium on Rendering (EGSR), 2021.
  * [5] S. Baker and Iain Matthews.  Lucas-kanade 20 years on: A unifying framework.  International Journal of Computer Vision (IJCV), 56:221–255, 2004\.
  * [6] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 5855–5864, 2021.
  * [7] Luca Bertinetto, Jack Valmadre, Stuart Golodetz, Ondrej Miksik, and Philip HS Torr.  Staple: Complementary learners for real-time tracking.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 1401–1409, 2016.
  * [8] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr.  Fully-convolutional siamese networks for object tracking.  In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 850–865, 2016.
  * [9] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch.  Nerd: Neural reflectance decomposition from image collections.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 12684–12694, 2021.
  * [10] Samuel Bryner, Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza.  Event-based, direct camera tracking from a photometric 3d map using nonlinear optimization.  In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), pages 325–331, 2019\.
  * [11] Rohan Chabra, Jan E Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe.  Deep local shapes: Learning local sdf priors for detailed 3d reconstruction.  In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 608–625, 2020.
  * [12] William Oswaldo Chamorro Hernandez, Juan Andrade-Cetto, and Joan Solà Ortega.  High-speed event camera tracking.  In Proc. Brit. Mach. Vis. Conf. (BMVC), pages 1–12, 2020.
  * [13] Che-Han Chang, Chun-Nan Chou, and Edward Y Chang.  Clkn: Cascaded lucas-kanade networks for image alignment.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 2213–2221, 2017.
  * [14] Shoushun Chen and Menghan Guo.  Live demonstration: Celex-v: A 1m pixel multi-mode event-based sensor.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), pages 1682–1683, 2019.
  * [15] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 5939–5948, 2019.
  * [16] Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam.  Blenderproc.  2019\.
  * [17] Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind.  Unconstrained scene generation with locally conditioned radiance fields.  Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2021.
  * [18] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 8649–8658, 2021.
  * [19] Guillermo Gallego, Jon EA Lund, Elias Mueggler, Henri Rebecq, Tobi Delbruck, and Davide Scaramuzza.  Event-based, 6-dof camera tracking from photometric depth maps.  IEEE Trans. Pattern Anal. Mach. Intell., 40(10):2402–2412, 2017\.
  * [20] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 14346–14355, 2021.
  * [21] Daniel Gehrig, Henri Rebecq, Guillermo Gallego, and Davide Scaramuzza.  Asynchronous, photometric feature tracking using events and frames.  In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 750–765, 2018.
  * [22] Daniel Gehrig, Henri Rebecq, Guillermo Gallego, and Davide Scaramuzza.  Eklt: Asynchronous photometric feature tracking using events and frames.  International Journal of Computer Vision (IJCV), 128(3):601–618, 2020.
  * [23] Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas.  BOP challenge 2020 on 6D object localization.  Proc. Eur. Conf. Comput. Vis. Workshops (ECCVW), pages 577–594, 2020.
  * [24] Jeffrey Ichnowski*, Yahav Avigal*, Justin Kerr, and Ken Goldberg.  Dex-NeRF: Using a neural radiance field to grasp transparent objects.  In Conference on Robot Learning (CoRL), 2020.
  * [25] Zhang Jiakai, Liu Xinhang, Ye Xinyi, Zhao Fuqiang, Zhang Yanshun, Wu Minye, Zhang Yingliang, Xu Lan, and Yu Jingyi.  Editable free-viewpoint video using a layered neural representation.  In SIGGRAPH, 2021.
  * [26] Ugur Kart, Alan Lukezic, Matej Kristan, Joni-Kristian Kamarainen, and Jiri Matas.  Object tracking by reconstruction with view-specific discriminative correlation filters.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 1339–1348, 2019.
  * [27] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs.  Neural human performer: Learning generalizable radiance fields for human performance rendering.  In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), volume 34, pages 24741–24752, 2021.
  * [28] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 8971–8980, 2018.
  * [29] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 6498–6508, 2021.
  * [30] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey.  Barf: Bundle-adjusting neural radiance fields.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 5741–5751, 2021.
  * [31] David B Lindell, Julien NP Martel, and Gordon Wetzstein.  Autoint: Automatic integration for fast neural volume rendering.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 14556–14565, 2021.
  * [32] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt.  Neural actor: Neural free-view synthesis of human actors with pose control.  ACM Trans. Graph. (TOG), 40(6):1–16, 2021.
  * [33] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell.  Editing conditional radiance fields.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 5773–5783, 2021.
  * [34] Bruce D. Lucas and Takeo Kanade.  An iterative image registration technique with an application to stereo vision.  In Proc. Int. Joint Conf. Artif. Intell. (IJCAI), volume 2, page 674–679, 1981.
  * [35] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 4460–4470, 2019.
  * [36] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM Trans. Graph. (TOG), 2019.
  * [37] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 405–421, 2020.
  * [38] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 11453–11464, 2021.
  * [39] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 5589–5599, 2021.
  * [40] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 165–174, 2019.
  * [41] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 5865–5874, 2021.
  * [42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), pages 8024–8035, 2019.
  * [43] Christoph Posch, Daniel Matolin, and Rainer Wohlgenannt.  High-dr frame-free pwm imaging with asynchronous aer intensity encoding and focal-plane temporal redundancy suppression.  In International Symposium on Circuits and Systems (ISCAS), pages 2430–2433, 2010.
  * [44] Prophesee metavision for machines.  https://www.prophesee.ai/.
  * [45] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 10318–10327, 2021.
  * [46] Henri Rebecq, Daniel Gehrig, and Davide Scaramuzza.  Esim: An open event camera simulator.  In Conference on Robot Learning (CoRL), pages 969–982, 2018.
  * [47] Henri Rebecq, Timo Horstschaefer, and Davide Scaramuzza.  Real-time visual-inertial odometry for event cameras using keyframe-based nonlinear optimization.  In Proc. Brit. Mach. Vis. Conf. (BMVC), 2017.
  * [48] Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022\.
  * [49] Teresa Serrano-Gotarredona and Bernabé Linares-Barranco.  A 128$\times$ 128 1.5% contrast sensitivity 0.9% fpn 3 $\mu$s latency 4 mw asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers.  IEEE Journal of Solid-State Circuits (JSSC), 48(3):827–838, 2013\.
  * [50] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), volume 33, pages 7462–7473, 2020.
  * [51] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 7495–7504, 2021.
  * [52] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison.  imap: Implicit mapping and positioning in real-time.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 6229–6238, 2021.
  * [53] Yunjae Suh, Seungnam Choi, Masamichi Ito, Jeongseok Kim, Youngho Lee, Jongseok Seo, Heejae Jung, Dong-Hee Yeo, Seol Namgung, Jongwoo Bong, et al.  A 1280$\times$ 960 dynamic vision sensor with a 4.95-$\mu$m pixel pitch and motion artifact minimization.  In International Symposium on Circuits and Systems (ISCAS), 2020\.
  * [54] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar.  Block-NeRF: Scalable large scene neural view synthesis.  2022\.
  * [55] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe.  Siam r-cnn: Visual tracking by re-detection.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 6578–6588, 2020.
  * [56] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2021.
  * [57] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  Nerf–: Neural radiance fields without known camera parameters.  2021\.
  * [58] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou.  Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 5610–5619, 2021.
  * [59] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 8534–8543, 2021.
  * [60] Christopher Xie, Keunhong Park, Ricardo Martin-Brualla, and Matthew Brown.  Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling.  In International Conference on 3D Vision (3DV), pages 962–971, 2021\.
  * [61] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui.  Learning object-compositional neural radiance field for editable scene rendering.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 13779–13788, 2021.
  * [62] Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin.  iNeRF: Inverting neural radiance fields for pose estimation.  In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), pages 1323–1330, 2021.
  * [63] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 5752–5761, 2021.
  * [64] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  ACM Trans. Graph. (TOG), 40(6):1–18, 2021.
  * [65] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison.  In-place scene labelling and understanding with implicit scene representation.  In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 15838–15847, 2021.
  * [66] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys.  Nice-slam: Neural implicit scalable encoding for slam.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 12786–12796, 2022.
  * [67] Alex Zihao Zhu, Nikolay Atanasov, and Kostas Daniilidis.  Event-based visual inertial odometry.  In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 5391–5399, 2017.

##  A NeRF

###  A.1 NeRF Formulation

NeRF [37] is an implicit 3D scene representation using neural networks,
originally proposed for novel view synthesis. As in Sec.3.1, NeRF,
$G_{\mathcal{M}}$, takes an image coordinate $(x,y)$ and 6DoF pose
$S\in\mathfrak{s}\mathfrak{e}(3)$ of the camera as inputs and render the RGB
intensity $\mathbf{c}$ at that coordinate. NeRF encodes a 3D scene as a
continuous representation using an MLP $f_{\mathcal{M}}$, parameterized by
learned parameter $\mathcal{M}$. The $f_{\mathcal{M}}$ output the intensity
$\mathbf{c}^{\prime}$ and volume density $\sigma$ given a viewing ray
$\mathbf{d}$ and a 3D coordinate $\mathbf{x}$. The viewing ray $\mathbf{d}$ is
computed from the camera pose $S$ and image coordinate $(x,y)$. Let the
homogeneous image coordinate $\bar{\mathbf{u}}=(x,y,1)$; then, the 3D point
$\mathbf{x}^{i}$ along the viewing ray $\mathbf{d}$ at depth $z^{i}$ is
expressed as $\mathbf{x}^{i}=\bar{\mathbf{u}}+z^{i}\mathbf{d}$. NeRF integrate
the intensity $\mathbf{c}^{\prime}$ using density $\sigma$ by volume rendering
to obtain intensity at the image coordinate $(x,y)$ for the given camera pose
$S$ as follows:

$\displaystyle G_{\mathcal{M}}(x,y,S)$
$\displaystyle=\int_{z_{n}}^{z_{f}}T(z)\sigma(\bar{\mathbf{u}}+z\mathbf{d})\mathbf{c}^{\prime}(\bar{\mathbf{u}}+z\mathbf{d})dz,$
(1) $\displaystyle T(z)$
$\displaystyle=\exp\left(-\int_{z_{n}}^{z}\sigma(\bar{\mathbf{u}}+z^{\prime}\mathbf{d})dz^{\prime}\right)$
(2)

where, $z_{n}$ and $z_{f}$ are bounds on the depth range of interest. In
practice, this rendering function is approximated numerically via quadrature
on points in the depth direction.

###  A.2 Network Architecture

The NeRF network architecture used for the experiments is shown in Fig.A. We
adopt code of NeRF from BARF official implementation
https://github.com/chenhsuanlin/bundle-adjusting-NeRF for training the model.
The difference from the original NeRF [37] is activation for density $\sigma$
is replaced by softplus from relu to improve stability during training. The
function $\gamma$ is a positional encoding function that maps inputs 3D
coordinates $x$ to higher dimensions of different sinusoidal frequency basis
functions.

Figure A: The NeRF network architecture used for the experiments.

##  B Event-based camera pose tracking dataset (EvTrack)

We created the 6DOF camera pose tracking dataset for event data, called
EvTrack, with ground-truth camera pose to demonstrate the effectiveness of the
proposed TeGRA and foster subsequent studies of event-based camera pose
tracking using implicit 3D scene representation.

We used BOP challenge 2020 scenes [23] because it is photo-realistic, and the
scene represents the indoor localization scenario well. The BOP 2020 uses the
BlenderProc [16] to render realistic images using ray-tracing. The dataset
consists of five scenes, mix, hb, lm, tyol and ycbb, the last four scenes
correspond to the BOP data split and mix scenes include all the four data to
simulate the ordinary indoor situation.

###  B.1 Data for training

The training data consists of 500 images with ground-truth camera poses for
each scene. Each camera pose is randomly sampled from a pre-defined
rectangular space that includes the tracking trajectories described in B.2.
The size of image $(H,W)$ is $(480\times 640)$. An implicit 3D scene
representation such as NeRF can be trained using the data.

###  B.2 Data for testing (tracking)

The test data consists of three camera trajectories for each scene simulating
the drone hovering around a room (Fig.3 right). For each scene, we render
1,000 consecutive images with ground-truth camera poses. The size of image
$(H,W)$ is $(480\times 640)$. Using the consecutive images, we generate IC-
event by subtracting the successive frames. The threshold $\delta$ for
triggering the event is set to $0.05$ (intensities are normalized to $[0,1]$)
for all scenes.

In our experiment, we use grayscale IC-event since most event cameras detect
grayscale intensity changes. To generate a grayscale IC-event, we convert the
RGB frame into a grayscale frame using OpenCV’s cvtColor() function.

###  B.3 Chunk size

The number of events consumed by the network at once (chunk size) must be
determined depending on the scene. In general, we can expect a higher accuracy
when using a longer chunk size as long as the chunked event stream conforms to
the motion model (we used the linear model of (5) in our experiment). On the
other hand, when the chunk size is too large, and the chunked event stream
contains non-linear motion, the observed event can not be explained by the
motion model, and the performance will degrade. We use four timestep
(generated from five consecutive frames) in our main experiment. The camera
motion in the EvTrack can be considered approximately linear within the five
frame interval.

###  B.4 Required accuracy for tracking

In the main paper, we discussed both dense approach and our sparse approach to
achieve comparable accuracy to keep the tracking (although their pose error
differs). The pose error from TeGRA is larger than the dense approach because
the observed event trajectory does not precisely conform to the motion model
we use (linear motion model), while the dense model aligns a single image
without temporal information (therefore, there is no model error). One could
utilize a non-linear motion model to reduce the error further. However, the
error from TeGRA is sufficiently small to keep stable tracking; therefore, the
linear model suffices for tracking (on the EvTrack dataset). Tab.B show the
tracking results on all the sequences in the EvTrack. We consider the pose is
successfully tracked ($\checkmark$) if the error in the final pose is less
than the initial pose error of $(1.0\degree,0.001)$ (Due to the time required
to run the dense RGB method (Fig.1), we used the randomly sampled 50
consecutive frames from each sequence).

We also confirmed that TeGRA could successfully track the camera pose without
drifting on the NeRF’s blender scene (supplement-F).

Table B: Tracking test of dense approach (Fig.1) and our sparse approach
(Fig.1, TeGRA).

|  | mix | hb | lm | tyol | ycbv
---|---|---|---|---|---|---
Dense | seq0 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
seq1 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
seq2 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
Sparse (ours) | seq0 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
seq1 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
seq2 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$

##  C Tracking using dense image

As discussed in the Sec.1 and (Fig.1), camera pose can be recovered using NeRF
and dense intensity observation. We use the same pre-trained NeRF model as our
TeGRA and estimate the camera pose $S$. Unlike ours, which optimizes pose $S$
and motion $\dot{S}$, the frame-based algorithm only optimizes pose $S$.
Because the entire $480\times 640$ pixels can not fit into the single GPU, we
split the pixels into a 4,000-pixel chunk and accumulated the gradient w.r.t.
$S$ to compute the single pose update for a frame. The pose update is repeated
$\operatorname{n\\_itr}$ (100) times for each observed frame. The learning
rate $\eta$ of the $S$ is set to $5\times 10^{-4}$ exponentially decaying to
$5\times 10^{-5}$ toward $\operatorname{n\\_itr}$.

##  D Binary Event to IC-Event

In this study, we use IC-event instead of binary-event, and synchronous event
stream instead of asynchronous one. We choose this experimental setup mainly
due to the implementational difficulties in generating event streams. It
requires engineering effort to generate asynchronous binary-event streams,
such as modifying event-camera simulators. Some event-based cameras can detect
the intensity change $e_{r}$ directly, such as Celex-V [14]. However, perhaps
the most common type reports an event
$e=[e_{x},e_{y},e_{t},e_{p}]^{\mathsf{T}}$ when it detects the intensity
change $\Delta L$ by a specified amount $\delta$, where $e_{p}\in[+1,-1]$ is
the polarity which indicates the increase or decrease of the intensity
changes.

We consider there are two possible approaches to make TeGRA compatible with
the binary-event; 1) Modify the loss function of (5) to adapt to the binary-
event (Sec.5.2), 2) Convert the binary-event to IC-event using the timestamp
(supplement-D). In this work, we use IC-event for simplicity and left the
exploration for future work.

In this section, we discuss the second approach. IC-event could be estimated
by a simple (linear fitting). One can use the time difference between the
latest two consecutive events on each pixel to estimate the intensity
difference, as depicted in Fig.D. There are other options, such as fitting
higher-order functions using more events.

Figure D: Binary-Event to IC-Event

##  E Algorithm of TeGRA

The complete algorithm of TeGRA is listed in Alg.E. Note that the algorithm
listed in Alg.E is slightly different from the one we used in our main
experiment (listing-LABEL:code). In the main experiments, we used a fixed
number of iterations for a fair comparison; however, we expect terminating
iteration using the norm of the pose & motion update is preferable for the
practical application in terms of computational efficiency.

Algorithm E TeGRA

0: IC-event stream $\mathbf{e}_{t}$Optimized pose & motion
$(S^{\mathrm{opt}}_{t-\nu},\dot{S}^{\mathrm{opt}}_{t-\nu})$ at time ${t-\nu}$

1: $\epsilon\leftarrow\inf$

2:
$({S}_{t}^{\mathrm{ini}},\dot{S}^{\mathrm{ini}}_{t})\leftarrow(S^{opt}_{t-\nu}+\nu\dot{S}^{\mathrm{opt}}_{t-\nu},\dot{S}^{\mathrm{opt}}_{t-\nu})$
$\triangleright{(\textrm{linear motion})}$

3:
$({S},{\dot{S}})\leftarrow(S^{\mathrm{ini}}_{t},\dot{S}^{\mathrm{ini}}_{t})$

4: while $\epsilon>\epsilon_{thr}$ do

5: Feedward event coordinate and pose to $G_{\mathcal{M}}$

6: Compute temporal gradient of $G_{\mathcal{M}}$ w.r.t
$[...,\bar{e}_{u}^{i},...]$

7: Evaluate error between $\nabla_{t}$NeRF($S_{t}$) and IC-event
$\triangleright{\text{Eq.(\ref{eq:theorem})}}$

8: Backpropagete the error thruough $\nabla_{t}$NeRF($S_{t}$)

9: Update pose & motion

10: $\epsilon\leftarrow$ norm of pose & motion update

11: end while

12: $({S}_{t}^{\mathrm{opt}},\dot{S}^{\mathrm{opt}}_{t})\leftarrow(S,\dot{S})$

12: $({S}_{t}^{\mathrm{opt}},\dot{S}^{\mathrm{opt}}_{t})$

##  F Additional tracking result on NeRF dataset

In addition to the main tracking experiment using EvTrack, we also conducted
an experiment to evaluate the tracking performance of TeGRA on the NeRF’s
blender scene [37]. For this experiment, we created the event stream from
1,000 consecutive frames using the blender scene; the camera rotates around
the object while moving up and down. We also confirmed that the camera pose is
successfully tracked without drift on this dataset (Fig.F, See also
supplemental video).

Figure F: Additional tracking results on NeRF’s blender scene.