###### Abstract

Contrast maximization (CMax) is a framework that provides state-of-the-art
results on several event-based computer vision tasks, such as ego-motion or
optical flow estimation. However, it may suffer from a problem called event
collapse, which is an undesired solution where events are warped into too few
pixels. As prior works have largely ignored the issue or proposed workarounds,
it is imperative to analyze this phenomenon in detail. Our work demonstrates
event collapse in its simplest form and proposes collapse metrics by using
first principles of space–time deformation based on differential geometry and
physics. We experimentally show on publicly available datasets that the
proposed metrics mitigate event collapse and do not harm well-posed warps. To
the best of our knowledge, regularizers based on the proposed metrics are the
only effective solution against event collapse in the experimental settings
considered, compared with other methods. We hope that this work inspires
further research to tackle more complex warp models.

###### keywords:

computer vision; intelligent sensors; robotics; event-based camera; contrast
maximization; optical flow; motion estimation

1 1 0 https://doi.org/ Event Collapse in Contrast Maximization Frameworks
Event Collapse in Contrast Maximization Frameworks Shintaro Shiba 1,2,* ,
Yoshimitsu Aoki 1 and Guillermo Gallego 2,3 Shintaro Shiba, Yoshimitsu Aoki
and Guillermo Gallego Shiba, S.; Aoki, Y.; Gallego, G. Correspondence:
<EMAIL_ADDRESS>
##  1 Introduction

Event cameras Delbruck (2008); Suh et al. (2020); Finateu et al. (2020) offer
potential advantages over standard cameras to tackle difficult scenarios (high
speed, high dynamic range, low power). However, new algorithms are needed to
deal with the unconventional type of data they produce (per-pixel asynchronous
brightness changes, called events) and unlock their advantages Gallego et al.
(2020). Contrast maximization (CMax) is an event processing framework that
provides state-of-the-art results on several tasks, such as rotational motion
estimation Gallego and Scaramuzza (2017); Kim and Kim (2021), feature flow
estimation and tracking Zhu et al. (2017a, b); Seok and Lim (2020); Stoffregen
and Kleeman (2019); Dardelet et al. (2021), ego-motion estimation Gallego et
al. (2018, 2019); Peng et al. (2021), 3D reconstruction Gallego et al. (2018);
Rebecq et al. (2018), optical flow estimation Zhu et al. (2019); Paredes-
Valles et al. (2019); Hagenaars et al. (2021); Shiba et al. (2022), motion
segmentation Mitrokhin et al. (2018); Stoffregen et al. (2019); Zhou et al.
(2021); Parameshwara et al. (2021); Lu et al. (2021), guided filtering Duan et
al. (2021), and image reconstruction Zhang et al. (2021).

The main idea of CMax and similar event alignment frameworks Nunes and Demiris
(2021); Gu et al. (2021) is to find the motion and/or scene parameters that
align corresponding events (i.e., events that are triggered by the same scene
edge), thus achieving motion compensation. The framework simultaneously
estimates the motion parameters and the correspondences between events (data
association). However, in some cases CMax optimization converges to an
undesired solution where events accumulate into too few pixels, a phenomenon
called event collapse (Figure 1). Because CMax is at the heart of many state-
of-the-art event-based motion estimation methods, it is important to
understand the above limitation and propose ways to overcome it. Prior works
have largely ignored the issue or proposed workarounds without analyzing the
phenomenon in detail. A more thorough discussion of the phenomenon is overdue,
which is the goal of this work.

Contrary to the expectation that event collapse occurs when the event
transformation becomes sufficiently complex Zhu et al. (2019); Nunes and
Demiris (2021), we show that it may occur even in the simplest case of one
degree-of-freedom (DOF) motion. Drawing inspiration from differential geometry
and electrostatics, we propose principled metrics to quantify event collapse
and discourage it by incorporating penalty terms in the event alignment
objective function. Although event collapse depends on many factors, our
strategy aims at modifying the objective’s landscape to improve the well-
posedness of the problem and be able to use well-known, standard optimization
algorithms.

|  |  |
---|---|---|---
Loss Landscape |  Original events at A |  Collapsed IWE at B |  Desired IWE at C
Figure 1: _Event Collapse._ Left: Landscape of the image variance loss as a
function of the warp parameter $h_{z}$. Right: The IWEs at the different
$h_{z}$ marked in the landspace. (A) Original events (identity warp),
accumulated over a small $\Delta t$ (polarity is not used). (B) Image of
warped events (IWE) showing event collapse due to maximization of the
objective function. (C) Desired IWE solution using our proposed regularizer:
sharper than (A) while avoiding event collapse (C).

In summary, our contributions are:

  1. 1.

A study of the event collapse phenomenon in regard to event warping and
objective functions (Sections 3.3 and 4).

  2. 2.

Two principled metrics of event collapse (one based on flow divergence and one
based on area-element deformations) and their use as regularizers to mitigate
the above-mentioned phenomenon (Sections 3.4 to 3.6).

  3. 3.

Experiments on publicly available datasets that demonstrate, in comparison
with other strategies, the effectiveness of the proposed regularizers (Section
4).

To the best of our knowledge, this is the first work that focuses on the
paramount phenomenon of event collapse, which may arise in state-of-the-art
event-alignment methods. Our experiments show that the proposed metrics
mitigate event collapse while they do not harm well-posed warps.

##  2 Related Work

###  2.1 Contrast Maximization

Our study is based on the CMax framework for event alignment (Figure 2, bottom
branch). The CMax framework is an iterative method with two main steps per
iteration: transforming events and computing an objective function from such
events. Assuming constant illumination, events are triggered by moving edges,
and the goal is to find the transformation/warping parameters
$\boldsymbol{\theta}$ (e.g., motion and scene) that achieve motion
compensation (i.e., alignment of events triggered at different times and
pixels), hence revealing the edge structure that caused the events. Standard
optimization algorithms (gradient ascent, sampling, etc.) can be used to
maximize the event-alignment objective. Upon convergence, the method provides
the best transformation parameters and the transformed events, i.e., motion-
compensated image of warped events (IWE).

The first step of the CMax framework transforms events according to a motion
or deformation model defined by the task at hand. For instance, camera
rotational motion estimation Gallego and Scaramuzza (2017); Liu et al. (2020)
often assumes constant angular velocity
($\boldsymbol{\theta}\equiv\boldsymbol{\omega}$) during short time spans,
hence events are transformed following 3-DOF motion curves defined on the
image plane by candidate values of $\boldsymbol{\omega}$. Feature tracking may
assume constant image velocity $\boldsymbol{\theta}\equiv\mathbf{v}$ (2-DOF)
Zhu et al. (2017a); Stoffregen and Kleeman (2017), hence events are
transformed following straight lines.

Figure 2: Proposed modification of the contrast maximization (CMax) framework
in Gallego et al. (2018, 2019) to also account for the degree of regularity
(collapsing behavior) of the warp. Events are colored in red/blue according to
their polarity. Reprinted/adapted with permission from Ref. Gallego et al.
(2019), 2019, Gallego et al.

In the second step of CMax, several event-alignment objectives have been
proposed to measure the goodness of fit between the events and the model
Gallego et al. (2019); Stoffregen and Kleeman (2019), establishing connections
between visual contrast, sharpness, and depth-from-focus. Finally, the choice
of iterative optimization algorithm also plays a big role in finding the
desired motion-compensation parameters. First-order methods, such as non-
linear conjugate gradient (CG), are a popular choice, trading off accuracy and
speed Gallego et al. (2018); Stoffregen et al. (2019); Zhou et al. (2021).
Exhaustive search, sampling, or branch-and-bound strategies may be affordable
for low-dimensional (DOF) search spaces Liu et al. (2020); Peng et al. (2021).
As will be presented (Section 3), our proposal consists of modifying the
second step by means of a regularizer (Figure 2, top branch).

###  2.2 Event Collapse

_In which estimation problems does event collapse appear?_ At first look, it
may appear that event collapse occurs when the number of DOFs in the warp
becomes large enough, i.e., for complex motions. Event collapse has been
reported in homographic motions (8 DOFs) Nunes and Demiris (2021); Ozawa et
al. (2022) and in dense optical flow estimation Zhu et al. (2019), where an
artificial neural network (ANN) predicts a flow field with $2N_{p}$ DOFs
($N_{p}$ pixels), whereas it does not occur in feature flow (2 DOFs) or
rotational motion flow (3 DOFs). However, a more careful analysis reveals that
this is not the entire story because event collapse may occur even in the case
of 1 DOF, as we show.

_How did previous works tackle event collapse?_ Previous works have tackled
the issue in several ways, such as: (i) initializing the parameters
sufficiently close to the desired solution (in the basin of attraction of the
local optimum) Gallego et al. (2018); (ii) reformulating the problem, changing
the parameter space to reduce the number of DOFs and increase the well-
posedness of the problem Ozawa et al. (2022); Peng et al. (2021); (iii)
providing additional data, such as depth Nunes and Demiris (2021), thus
changing the problem from motion estimation given only events to motion
estimation given events and additional sensor data; (iv) whitening the warped
events before computing the objective Nunes and Demiris (2021); and (v)
redesigning the objective function and possibly adding a strong classical
regularizer (e.g., Charbonnier loss) Zhu et al. (2019); Stoffregen and Kleeman
(2019). Many of the above mitigation strategies are task-specific because it
may not always be possible to consider additional data or reparametrize the
estimation problem. Our goal is to approach the issue without the need for
additional data or changing the parameter space, and to show how previous
objective functions and newly regularized ones handle event collapse.

##  3 Method

Let us present our approach to measure and mitigate event collapse. First, we
revise how event cameras work (Section 3.1) and the CMax framework (Section
3.2), which was informally introduced in Section 2.1. Then, Section 3.3 builds
our intuition on event collapse by analyzing a simple example. Section 3.4
presents our proposed metrics for event collapse, based on 1-DOF and 2-DOF
warps. Section 3.5 specifies them for higher DOFs, and Section 3.6 presents
the regularized objective function.

###  3.1 How Event Cameras Work

Event cameras, such as the Dynamic Vision Sensor (DVS) Lichtsteiner et al.
(2008); Suh et al. (2020); Finateu et al. (2020), are bio-inspired sensors
that capture pixel-wise intensity changes, called events, instead of intensity
images. An event $e_{k}\doteq(\mathbf{x}_{k},t_{k},p_{k})$ is triggered as
soon as the logarithmic intensity $L$ at a pixel exceeds a contrast
sensitivity $C>0$,

$L(\mathbf{x}_{k},t_{k})-L(\mathbf{x}_{k},t_{k}-\Delta t_{k})=p_{k}\,C,$ (1)

where $\mathbf{x}_{k}\doteq(x_{k},y_{k})^{\top}$, $t_{k}$ (with
$\mathrm{\SIUnitSymbolMicro s}$ resolution) and polarity $p_{k}\in\\{+1,-1\\}$
are the spatio-temporal coordinates and sign of the intensity change,
respectively, and $t_{k}-\Delta t_{k}$ is the time of the previous event at
the same pixel $\mathbf{x}_{k}$. Hence, each pixel has its own sampling rate,
which depends on the visual input.

###  3.2 Mathematical Description of the CMax Framework

The CMax framework Gallego et al. (2018) transforms events in a set
$\mathcal{E}=\\{e_{k}\\}_{k=1}^{N_{e}}$ geometrically

$e_{k}\doteq(\mathbf{x}_{k},t_{k},p_{k})\quad\stackrel{{\scriptstyle\mathbf{W}}}{{\mapsto}}\quad
e^{\prime}_{k}\doteq(\mathbf{x}^{\prime}_{k},t_{\text{ref}},p_{k}),$ (2)

according to a motion model $\mathbf{W}$, producing a set of warped events
$\mathcal{E}^{\prime}=\\{e^{\prime}_{k}\\}_{k=1}^{N_{e}}$. The warp
$\mathbf{x}^{\prime}_{k}=\mathbf{W}(\mathbf{x}_{k},t_{k};\boldsymbol{\theta})$
transports each event along the point trajectory that passes through it
(Figure 2, left), until $t_{\text{ref}}$ is reached. The point trajectories
are parametrized by $\boldsymbol{\theta}$, which contains the motion and/or
scene unknowns. Then, an objective function Gallego et al. (2019); Stoffregen
and Kleeman (2019) measures the alignment of the warped events
$\mathcal{E}^{\prime}$. Many objective functions are given in terms of the
count of events along the point trajectories, which is called the image of
warped events (IWE):

$I(\mathbf{x};\boldsymbol{\theta})\doteq\sum_{k=1}^{N_{e}}b_{k}\,\delta(\mathbf{x}-\mathbf{x}^{\prime}_{k}(\boldsymbol{\theta})).$
(3)

Each IWE pixel $\mathbf{x}$ sums the values of the warped events
$\mathbf{x}^{\prime}_{k}$ that fall within it: $b_{k}=p_{k}$ if polarity is
used or $b_{k}=1$ if polarity is not used. The Dirac delta $\delta$ is in
practice replaced by a smooth approximation Ng et al. (2022), such as a
Gaussian,
$\delta(\mathbf{x}-\boldsymbol{\mu})\approx\mathcal{N}(\mathbf{x};\boldsymbol{\mu},\epsilon^{2})$
with $\epsilon=1$ pixel. A popular objective function $G(\boldsymbol{\theta})$
is the visual contrast of the IWE (3), given by the variance

$G(\boldsymbol{\theta})\equiv\operatorname{Var}\bigl{(}I(\mathbf{x};\boldsymbol{\theta})\bigr{)}\doteq\frac{1}{|\Omega|}\int_{\Omega}(I(\mathbf{x};\boldsymbol{\theta})-\mu_{I})^{2}d\mathbf{x},$
(4)

with mean
$\mu_{I}\doteq\frac{1}{|\Omega|}\int_{\Omega}I(\mathbf{x};\boldsymbol{\theta})d\mathbf{x}$
and image domain $\Omega$. Hence, the alignment of the transformed events
$\mathcal{E}^{\prime}$ (i.e., the candidate “corresponding events”, triggered
by the same scene edge) is measured by the strength of the edges of the IWE.
Finally, an optimization algorithm iterates the above steps until the best
parameters are found:

$\boldsymbol{\theta}^{\ast}=\arg\max_{\boldsymbol{\theta}}G(\boldsymbol{\theta}).$
(5)

###  3.3 Simplest Example of Event Collapse: 1 DOF

To analyze event collapse in the simplest case, let us consider an
approximation to a translational motion of the camera along its optical axis
$Z$ (1-DOF warp). In theory, translational motions also require the knowledge
of the scene depth. Here, inspired by the 4-DOF in-plane warp in Mitrokhin et
al. (2018) that approximates a 6-DOF camera motion, we consider a simplified
warp that does not require knowledge of the scene depth. In terms of data, let
us consider events from one of the driving sequences of the standard MVSEC
dataset Zhu et al. (2018) (Figure 1).

For further simplicity, let us normalize the timestamps of $\mathcal{E}$ to
the unit interval $t\in[t_{1},t_{N_{e}}]\mapsto\tilde{t}\in[0,1]$, and assume
a coordinate frame at the center of the image plane, then the warp
$\mathbf{W}$ is given by

$\mathbf{x}^{\prime}_{k}=(1-\tilde{t}_{k}h_{z})\,\mathbf{x}_{k},$ (6)

where $\boldsymbol{\theta}\equiv h_{z}$. Hence, events are transformed along
the radial direction from the image center, acting as a virtual focus of
expansion (FOE) (cf. the true FOE is given by the data). Letting the scaling
factor in (6) be $s_{k}\doteq 1-\tilde{t}_{k}h_{z}$, we observe the following:
(i) $s_{k}$ cannot be negative since it would imply that at least one event
has flipped the side on which it lies with respect to the image center; (ii)
if $s_{k}>1$ the warped event gets away from the image center (“expansion” or
“zoom-in”); and (iii) if $s_{k}\in[0,1)$ the warped event gets closer to the
image center (“contraction” or “zoom-out”). The equivalent conditions in terms
of $h_{z}$ are: (i) $h_{z}<1$, (ii) $h_{z}<0$ is an expansion, and (iii)
$0<h_{z}<1$ is a contraction.

Intuitively, event collapse occurs if the contraction is large ($0<s_{k}\ll
1$) (see Figures 1C and 3a). This phenomenon is not specific of the image
variance; other objective functions lead to the same result. As we see, the
objective function has a local maximum at the desired motion parameters
(Figure 1B). The optimization over the entire parameter space converges to a
global optimum that explains the event collapse.

a

b

c

Figure 3: _Point trajectories_ (streamlines) defined on $x-y-t$ image space by
various warps. (a) Zoom in/out warp from image center (1 DOF). (b) Constant
image velocity warp (2 DOF). (c) Rotational warp around $X$ axis (3 DOF).

#### Discussion

The above example shows that event collapse is enabled (or disabled) by the
type of warp. If the warp does not enable event collapse (contraction or
accumulation of flow vectors cannot happen due to the geometric properties of
the warp), as in the case of feature flow (2 DOF) Zhu et al. (2017a);
Stoffregen and Kleeman (2017) (Figure 3b) or rotational motion flow (3 DOF)
Gallego and Scaramuzza (2017); Liu et al. (2020) (Figure 3c), then the
optimization problem is well posed and multiple objective functions can be
designed to achieve event alignment Gallego et al. (2019); Stoffregen and
Kleeman (2019). However, the disadvantage is that the type of warps that
satisfy this condition may not be rich enough to describe complex scene
motions.

On the other hand, if the warp allows for event collapse, more complex
scenarios can be described by such a broader class of motion hypotheses, but
the optimization framework designed for non-event-collapsing scenarios (where
the local maximum is assumed to be the global maximum) may not hold anymore.
Optimizing the objective function may lead to an undesired solution with a
larger value than the desired one. This depends on multiple elements: the
landscape of the objective function (which depends on the data, the warp
parametrization, and the shape of the objective function), and the
initialization and search strategy of the optimization algorithm used to
explore such a landscape. The challenge in this situation is to overcome the
issue of multiple local maxima and make the problem better posed. Our approach
consists of characterizing event collapse via novel metrics and including them
in the objective function as weak constraints (penalties) to yield a better
landscape.

###  3.4 Proposed Regularizers

####  3.4.1 Divergence of the Event Transformation Flow

Inspired by physics, we may think of the flow vectors given by the event
transformation $\mathcal{E}\mapsto\mathcal{E}^{\prime}$ as an electrostatic
field, whose sources and sinks correspond to the location of electric charges
(Figure 4). Sources and sinks are mathematically described by the divergence
operator $\nabla\cdot\,$. Therefore, the divergence of the flow field is a
natural choice to characterize event collapse.

Figure 4: _Divergence of different vector fields_ ,
$\nabla\cdot\mathbf{v}=\partial_{x}\mathbf{v}_{x}+\partial_{y}\mathbf{v}_{y}$.
From left to right: contraction (“sink”, leading to event collapse), expansion
(“source”), and incompressible fields. Image adapted from khanacademy.org

The warp $\mathbf{W}$ is defined over the space-time coordinates of the
events, hence its time derivative defines a flow field over space-time:

$\mathbf{f}\doteq\frac{\partial\mathbf{W}(\mathbf{x},t;\boldsymbol{\theta})}{\partial
t}.$ (7)

For the warp in (6), we obtain $\mathbf{f}=-h_{z}\mathbf{x}$, which gives
$\nabla\cdot\mathbf{f}=-h_{z}\nabla\cdot\mathbf{x}=-2h_{z}$. Hence, (6)
defines a constant divergence flow, and imposing a penalty on the degree of
concentration of the flow field accounts to directly penalizing the value of
the parameter $h_{z}$.

Computing the divergence at each event gives the set

$\mathcal{D}(\mathcal{E},\boldsymbol{\theta})\doteq\\{\nabla\cdot\mathbf{f}_{k}\\}_{k=1}^{N_{e}},$
(8)

from which we can compute statistical scores (mean, median, $\min$, etc.):

$\displaystyle R_{D}(\mathcal{E},\boldsymbol{\theta})$
$\displaystyle\doteq\frac{1}{N_{e}}\sum_{k=1}^{N_{e}}\nabla\cdot\mathbf{f}_{k}.$
$\displaystyle(\text{mean})$ (9)

To have a 2D visual representation (“feature map”) of collapse, we build an
image (like the IWE) by taking some statistic of the values
$\nabla\cdot\mathbf{f}_{k}$ that warp to each pixel, such as the “average
divergence per pixel”:

$\text{DIWE}(\mathbf{x};\mathcal{E},\boldsymbol{\theta})\doteq\frac{1}{N_{e}(\mathbf{x})}\sum_{k}(\nabla\cdot\mathbf{f}_{k})\,\delta(\mathbf{x}-\mathbf{x}^{\prime}_{k}),$
(10)

where
$N_{e}(\mathbf{x})\doteq\sum_{k}\delta(\mathbf{x}-\mathbf{x}^{\prime}_{k})$ is
the number of warped events at pixel $\mathbf{x}$ (the IWE). Then we aggregate
further into a score, such as the mean:

$R_{\text{DIWE}}(\mathcal{E},\boldsymbol{\theta})\doteq\frac{1}{|{\Omega}|}\int_{\Omega}\text{DIWE}(\mathbf{x};\mathcal{E},\boldsymbol{\theta})d\mathbf{x}.$
(11)

In practice we focus on the collapsing part by computing a trimmed mean: the
mean of the DIWE pixels smaller than a margin $\alpha$ ($-0.2$ in the
experiments). Such a margin does not penalize small, admissible deformations.

####  3.4.2 Area-Based Deformation of the Event Transformation

In addition to vector calculus, we may also use tools from differential
geometry to characterize event collapse. Building on Gallego et al. (2018),
the point trajectories define the streamlines of the transformation flow, and
we may measure how they concentrate or disperse based on how the area element
deforms along them. That is, we consider a small area element $dA=dxdy$
attached to each point along the trajectory and measure how much it deforms
when transported to the reference time: $dA^{\prime}=|\det(\mathtt{J})|\,dA$,
with the Jacobian

$\mathtt{J}(\mathbf{x},t;\boldsymbol{\theta})\doteq\frac{\partial\mathbf{W}(\mathbf{x},t;\boldsymbol{\theta})}{\partial\mathbf{x}}$
(12)

(see Section 5). The determinant of the Jacobian is the amplification factor:
$|\det(\mathtt{J})|>1$ if the area expands, and $|\det(\mathtt{J})|<1$ if the
area shrinks.

|  |
---|---|---
Contraction |  No change of area |  Expansion
Figure 5:  _Area deformation of various warps_. An area of
$dA\leavevmode\nobreak\ \text{pix}^{2}$ at $(\mathbf{x}_{k},t_{k})$ and is
warped to $t_{\text{ref}}$, giving an area
$dA^{\prime}=|\det(\mathtt{J}_{k})|dA\leavevmode\nobreak\ \text{pix}^{2}$ at
$(\mathbf{x}^{\prime}_{k},t_{\text{ref}})$, where
$\mathtt{J}_{k}\equiv\mathtt{J}(e_{k})\equiv\mathtt{J}(\mathbf{x}_{k},t_{k};\boldsymbol{\theta})$
(see (12)). From left to right, increasing area amplification factor
$|\det(\mathtt{J})|\in[0,\infty)$.

For the warp in (6), we have the Jacobian
$\mathtt{J}=(1-\tilde{t}h_{z})\mathtt{Id}$, and so
$\det(\mathtt{J})=(1-\tilde{t}h_{z})^{2}$. Interestingly, the area deformation
around event $e_{k}$,
$\mathtt{J}(e_{k})\equiv\mathtt{J}(\mathbf{x}_{k},t_{k};\boldsymbol{\theta})$,
is directly related to the scaling factor $s_{k}$:
$\det(\mathtt{J}(e_{k}))=s_{k}^{2}$.

Computing the amplification factors at each event gives the set

$\mathcal{A}(\mathcal{E},\boldsymbol{\theta})\doteq\bigl{\\{}|\det(\mathtt{J}(e_{k}))|\bigr{\\}}_{k=1}^{N_{e}},$
(13)

from which we can compute statistical scores. For example,

$\displaystyle R_{A}(\mathcal{E},\boldsymbol{\theta})$
$\displaystyle\doteq\frac{1}{N_{e}}\sum_{k=1}^{N_{e}}|\det(\mathtt{J}(e_{k}))|$
$\displaystyle(\text{mean})$ (14)

gives an average score: $R_{A}>1$ for expansion, and $R_{A}<1$ for
contraction.

We build a deformation map (or image of warped areas (IWA)) by taking some
statistic of the values $|\det(\mathtt{J}(e_{k}))|$ that warp to each pixel,
such as the “average amplification per pixel”:

$\text{IWA}(\mathbf{x})\doteq
1+\frac{1}{N_{e}(\mathbf{x})}\sum_{k=1}^{N_{e}}\bigl{(}|\det(\mathtt{J}(e_{k}))|-1\bigr{)}\,\delta(\mathbf{x}-\mathbf{x}^{\prime}_{k}).$
(15)

This assumes that if no events warp to a pixel $\mathbf{x}_{p}$, then
$N_{e}(\mathbf{x}_{p})=0$, and there is no deformation
($\text{IWA}(\mathbf{x}_{p})=1$). Then, we summarize the deformation map into
a score, such as the mean:

$R_{\text{IWA}}(\mathcal{E},\boldsymbol{\theta})\doteq\frac{1}{|{\Omega}|}\int_{\Omega}\text{IWA}(\mathbf{x};\mathcal{E},\boldsymbol{\theta})d\mathbf{x}.$
(16)

To concentrate on the collapsing part, we compute a trimmed mean: the mean of
the IWA pixels smaller than a margin $\alpha$ ($0.8$ in the experiments). The
margin approves small, admissible deformations.

###  3.5 Higher DOF Warp Models

####  3.5.1 Feature Flow

Event-based feature tracking is often described by the warp
$\mathbf{W}(\mathbf{x},t;\boldsymbol{\theta})=\mathbf{x}+(t-t_{\text{ref}})\boldsymbol{\theta}$,
which assumes constant image velocity $\boldsymbol{\theta}$ (2 DOFs) over
short time intervals. As expected, the flow for this warp coincides with the
image velocity, $\mathbf{f}=\boldsymbol{\theta}$, which is independent of the
space-time coordinates ($\mathbf{x},t$). Hence, the flow is incompressible
($\nabla\cdot\mathbf{f}=\mathbf{0}$): the streamlines given by the feature
flow do not concentrate or disperse; they are parallel. Regarding the area
deformation, the Jacobian
$\mathtt{J}=\partial(\mathbf{x}+(t-t_{\text{ref}})\boldsymbol{\theta})/\partial\mathbf{x}=\mathtt{Id}$
is the identity matrix. Hence $|\det(\mathtt{J})|=1$, that is, translations on
the image plane do not change the area of the pixels around a point.

In-plane translation warps, such as the above 2-DOF warp, are well-posed and
serve as reference to design the regularizers that measure event collapse. It
is sensible for well-designed regularizers to penalize warps whose
characteristics deviate from those of the reference warp: zero divergence and
unit area amplification factor.

####  3.5.2 Rotational Motion

As the previous sections show, the proposed metrics designed for the zoom
in/out warp produce the expected characterization of the 2-DOF feature flow
(zero divergence and unit area amplification), which is a well-posed warp.
Hence, if they were added as penalties into the objective function they would
not modify the energy landscape. We now consider their influence on rotational
motions, which are also well-posed warps. In particular, we consider the
problem of estimating the angular velocity of a predominantly rotating event
camera by means of CMax, which is a popular research topic Gallego and
Scaramuzza (2017); Liu et al. (2020); Peng et al. (2021); Nunes and Demiris
(2021); Gu et al. (2021). By using calibrated and homogeneous coordinates, the
warp is given by

$\mathbf{x}^{h\prime}\sim\mathtt{R}(t\boldsymbol{\omega})\,\mathbf{x}^{h},$
(17)

where
$\boldsymbol{\theta}\equiv\boldsymbol{\omega}=(\omega_{1},\omega_{2},\omega_{3})^{\top}$
is the angular velocity, $t\in[0,\Delta t]$, and $\mathtt{R}$ is parametrized
by using exponential coordinates (Rodrigues rotation formula Murray et al.
(1994); Gallego and Yezzi (2014)).

Divergence: It is well known that the flow is
$\mathbf{f}=B(\mathbf{x})\,\boldsymbol{\omega}$, where $B(\mathbf{x})$ is the
rotational part of the feature sensitivity matrix Corke (2017). Hence

$\nabla\cdot\mathbf{f}=3(x\omega_{2}-y\omega_{1}).$ (18)

Area element: Letting $\mathbf{r}_{3}^{\top}$ be the third row of
$\mathtt{R}$, and using (32)–(34) in Gallego et al. (2011),

$\det(\mathtt{J})=(\mathbf{r}_{3}^{\top}\mathbf{x}^{h})^{-3}.$ (19)

Rotations around the $Z$ axis clearly present no deformation, regardless of
the amount of rotation, and this is captured by the proposed metrics because:
(i) the divergence is zero, thus the flow is incompressible, and (ii)
$\det(\mathtt{J})=1$ since $\mathbf{r}_{3}=(0,0,1)^{\top}$ and
$\mathbf{x}^{h}=(x,y,1)^{\top}$. For other, arbitrary rotations, there are
deformations, but these are mild if the rotation angle $\Delta
t\|\boldsymbol{\omega}\|$ is small.

####  3.5.3 Planar Motion

Planar motion is the term used to describe the motion of a ground robot that
can translate and rotate freely on a flat ground. If such a robot is equipped
with a camera pointing upwards or downwards, the resulting motion induced on
the image plane, parallel to the ground plane, is an isometry (Euclidean
transformation). This motion model is a subset of the parametric ones in
Gallego et al. (2018), and it has been used for CMax in Peng et al. (2021);
Nunes and Demiris (2021). For short time intervals, planar motion may be
parametrized by 3 DOFs: linear velocity (2 DOFs) and angular velocity (1 DOF).
As the divergence and area metrics show in the Appendix, planar motion is a
well-posed warp. The resulting motion curves on the image plane do not lead to
event collapse.

####  3.5.4 Similarity Transformation

The 1-DOF zoom in/out warp in Section 3.3 is a particular case of the 4-DOF
warp in Mitrokhin et al. (2018), which is an in-plane approximation to the
motion induced by a freely moving camera. The same idea of combining
translation, rotation, and scaling for CMax is expressed by the similarity
transformation in Nunes and Demiris (2021). Both 4-DOF warps enable event
collapse because they allow for zoom-out motion curves. Formulas justifying it
are given in the Appendix.

###  3.6 Augmented Objective Function

We propose to augment previous objective functions (e.g., (5)) with penalties
obtained from the metrics developed above for event collapse:

$\boldsymbol{\theta}^{\ast}=\arg\\!\min_{\boldsymbol{\theta}}J(\boldsymbol{\theta})=\arg\\!\min_{\boldsymbol{\theta}}\left(-G(\boldsymbol{\theta})+\lambda
R(\boldsymbol{\theta})\right).$ (20)

We may interpret $G(\boldsymbol{\theta})$ (e.g., contrast or focus score
Gallego et al. (2019)) as the data fidelity term and $R(\boldsymbol{\theta})$
as the regularizer, or, in Bayesian terms, the likelihood and the prior,
respectively.

##  4 Experiments

We evaluate our method on publicly available datasets, whose details are
described in Section 4.1. First, Section 4.2 shows that the proposed
regularizers mitigate the overfitting issue on warps that enable collapse. For
this purpose we use driving datasets (MVSEC Zhu et al. (2018), DSEC Gehrig et
al. (2021)). Next, Section 4.3 shows that the regularizers do not harm well-
posed warps. To this end, we use the ECD dataset Mueggler et al. (2017).
Finally, Section 4.4 conducts a sensitivity analysis of the regularizers.

###  4.1 Evaluation Datasets and Metrics

####  4.1.1 Datasets

The _MVSEC_ dataset Zhu et al. (2018) is a widely used dataset for various
vision tasks, such as optical flow estimation Zhu et al. (2019); Gehrig et al.
(2021); Nagata et al. (2021); Hagenaars et al. (2021); Shiba et al. (2022).
Its sequences are recorded on a drone (indoors) or on a car (outdoors), and
comprise events, grayscale frames and IMU data from an mDAVIS346 Taverni et
al. (2018) ($346\times 260$ pixels), as well as camera poses and LiDAR data.
Ground truth optical flow is computed as the motion field Zhu et al. (2018),
given the camera velocity and the depth of the scene (from the LiDAR). We
select several excerpts from the _outdoor_day1_ sequence with a forward
motion. This motion is reasonably well approximated by collapse-enabled warps
such as (6). In total, we evaluate 3.2 million events spanning 10 s.

The _DSEC_ dataset Gehrig et al. (2021) is a more recent driving dataset with
a higher resolution event camera (Prophesee Gen3, $640\times 480$ pixels).
Ground truth optical flow is also computed as the motion field using the scene
depth from a LiDAR Gehrig et al. (2021). We evaluate on the _zurich_city_11_
sequence, using in total 380 million events spanning 40 s.

The _ECD_ dataset Mueggler et al. (2017) is the de facto standard to assess
event camera ego-motion Gallego and Scaramuzza (2017); Zhu et al. (2017b);
Rosinol Vidal et al. (2018); Gu et al. (2021); Rebecq et al. (2017); Mueggler
et al. (2018); Zhou et al. (2021). Each sequence provides events, frames, a
calibration file, and IMU data (at 1kHz) from a DAVIS240C camera Brandli et
al. (2014) ($240\times 180$ pixels), as well as ground-truth camera poses from
a motion-capture system (at 200Hz). For rotational motion estimation (3DOF),
we use the natural-looking _boxes_rotation_ and _dynamic_rotation_ sequences.
We evaluate 43 million events (10 s) of the box sequence, and 15 million
events (11 s) of the dynamic sequence.

The driving datasets (MVSEC, DSEC) and the selected sequences in the ECD
dataset have different type of motions: forward (which enables event collapse)
vs. rotational (which does not suffer from event collapse). Each sequence
serves a different test purpose, as discussed in the next sections.

####  4.1.2 Metrics

The metrics used to assess optical flow accuracy (MVSEC and DSEC datasets) are
the average endpoint error (AEE) and the percentage of pixels with AEE greater
than $N$ pixels (denoted by “$N$PE”, for $N=\\{3,10,20\\}$). Both are measured
over pixels with valid ground-truth values. We also use the FWL metric
Stoffregen et al. (2020) to assess event alignment by means of the IWE
sharpness (the FWL is the IWE variance relative to that of the identity warp).

Following previous works Gallego et al. (2019); Nunes and Demiris (2021); Gu
et al. (2021), rotational motion accuracy is assessed as the RMS error of
angular velocity estimation. Angular velocity $\boldsymbol{\omega}$ is assumed
to be constant over a window of events, estimated and compared with the ground
truth at the midpoint of the window. Additionally, we use the FWL metric to
gauge event alignment Stoffregen et al. (2020).

The event time windows are as follows: the events in the time spanned by
$dt=4$ frames in MVSEC (standard in Zhu et al. (2019); Gehrig et al. (2021);
Hagenaars et al. (2021)), 500k events for DSEC, and 30k events for ECD Gu et
al. (2021). The regularizer weights for divergence ($\lambda_{\text{div}}$)
and deformation ($\lambda_{\text{def}}$) are as follows:
$\lambda_{\text{div}}=2$ and $\lambda_{\text{def}}=5$ for MVSEC,
$\lambda_{\text{div}}=50$ and $\lambda_{\text{def}}=100$ for DSEC, and
$\lambda_{\text{div}}=5$ and $\lambda_{\text{def}}=10$ for ECD experiments.

###  4.2 Effect of the Regularizers on Collapse-Enabled Warps

Tables 1 and 2 report the results on the MVSEC and DSEC benchmarks,
respectively, by using two different loss functions $G$: the IWE variance (4)
and the squared magnitude of the IWE gradient, abbreviated “Gradient
Magnitude” Gallego et al. (2019). For MVSEC, we report the accuracy within the
time interval of $dt=4$ grayscale frame (at $\approx$ 45Hz). The optimization
algorithm is the Tree-Structured Parzen Estimator (TPE) sampler Bergstra et
al. (2011) for both experiments, with a number of sampling points equal to 300
(1 DOF) and 600 (4 DOF). The tables quantitatively capture the collapse
phenomenon suffered by the original CMax framework Gallego et al. (2018) and
the whitening technique Nunes and Demiris (2021). Their high FWL values
indicate that contrast is maximized; however, the AEE and $N$PE values are
exceedingly high (e.g., $>80$ pixels, $20\text{PE}>80$%), indicating that the
estimated flow is unrealistic.

Table 1: Results of MVSEC dataset Zhu et al. (2018).

Variance Gradient Magnitude AEE $\boldsymbol{\downarrow}$ 3PE
$\boldsymbol{\downarrow}$ 10PE $\boldsymbol{\downarrow}$ 20PE
$\boldsymbol{\downarrow}$ FWL $\boldsymbol{\uparrow}$ AEE
$\boldsymbol{\downarrow}$ 3PE $\boldsymbol{\downarrow}$ 10PE
$\boldsymbol{\downarrow}$ 20PE $\boldsymbol{\downarrow}$ FWL
$\boldsymbol{\uparrow}$ Ground truth flow _ _ _ _ $1.047\,993\,099$ _ _ _ _
$1.047\,993\,099$ Identity warp $4.845\,102\,657$ $60.585\,450\,33$
$10.384\,180\,53$ $0.314\,555\,551$ $1.0$ $4.845\,102\,657$ $60.585\,450\,33$
$10.384\,180\,53$ $0.314\,555\,551$ $1.0$ 1 DOF No regularizer $89.339\,76$
$97.299\,11$ $95.417\,347\,07$ $92.390\,566\,33$ $1.904\,380\,4$ $85.774\,515$
$93.964\,33$ $86.235\,230\,74$ $83.445\,622\,01$ $1.8692$ Whitening Nunes and
Demiris (2021) $89.5848$ $97.181\,28$ $96.772\,802\,41$ $93.760\,300\,44$
$1.9013$ $81.101\,769\,19$ $90.860\,110\,26$ $89.043\,858\,9$
$86.200\,254\,93$ $1.845\,830\,887$ Divergence (Ours) $3.997\,768$
$46.019\,21$ $2.771\,386\,599$ $0.054\,922\,492\,24$ $1.119\,496$
$2.872\,214\,97$ $32.678\,427\,11$ $2.516\,948\,313$ $0.031\,582\,055\,48$
$1.173\,016\,021$ Deformation (Ours) $4.465\,84$ $52.600\,07$
$5.162\,230\,828$ $0.127\,030\,408$ $1.0766$ $3.970\,13$ $48.7886$
$3.209\,162\,316$ $0.066\,383\,923\,14$ $1.0873$ Div. + Def. (Ours)
$3.299\,381$ $33.087\,535\,45$ $2.613\,900\,583$ $0.484\,262\,705\,5$
$1.196\,874\,476$ $2.849\,103\,044$ $32.337\,056\,58$ $2.435\,868\,209$
$0.032\,464\,283\,93$ $1.174\,927\,461$ 4 DOF Mitrokhin et al. (2018) No
regularizer $90.215\,397\,7$ $90.215\,397\,7$ $96.940\,675\,2$
$93.864\,536\,51$ $2.046\,721\,054$ $91.263\,285\,56$ $99.485\,319\,96$
$95.061\,646\,99$ $91.463\,053\,09$ $2.013\,541\,325$ Whitening Nunes and
Demiris (2021) $90.818\,409\,15$ $99.109\,810\,7$ $98.037\,577\,81$
$95.041\,879\,6$ $2.039\,896\,952$ $88.384\,647\,66$ $98.869\,765\,57$
$92.411\,993\,18$ $88.663\,739\,37$ $1.999\,155\,093$ Divergence (Ours)
$7.254\,550\,846$ $81.747\,412\,26$ $18.531\,339\,53$ $0.692\,149\,674\,3$
$1.086\,032\,753$ $5.365\,312\,689$ $66.179\,470\,36$ $10.809\,873\,21$
$0.276\,475\,703\,2$ $1.143\,905\,64$ Deformation (Ours) $8.132\,825\,707$
$87.459\,245\,6$ $18.531\,339\,53$ $1.092\,239\,517$ $1.027\,891\,481$
$5.253\,162\,047$ $64.787\,401\,94$ $13.175\,004\,67$ $0.373\,906\,813\,8$
$1.151\,436\,025$ Div. + Def. (Ours) $5.144\,825\,74$ $65.607\,465\,89$
$10.754\,696\,45$ $0.381\,769\,796\,2$ $1.157\,302\,645$ $5.409\,725\,124$
$66.014\,200\,62$ $13.188\,804\,75$ $0.541\,780\,113\,6$ $1.143\,399\,47$

Table 2: Results of DSEC dataset Gehrig et al. (2021).

Variance Gradient Magnitude AEE $\boldsymbol{\downarrow}$ 3PE
$\boldsymbol{\downarrow}$ 10PE $\boldsymbol{\downarrow}$ 20PE
$\boldsymbol{\downarrow}$ FWL $\boldsymbol{\uparrow}$ AEE
$\boldsymbol{\downarrow}$ 3PE $\boldsymbol{\downarrow}$ 10PE
$\boldsymbol{\downarrow}$ 20PE $\boldsymbol{\downarrow}$ FWL
$\boldsymbol{\uparrow}$ Ground truth flow _ _ _ _ $1.090\,988\,475$ _ _ _ _
$1.090\,988\,475$ Identity warp $5.843\,080\,683$ $60.453\,759\,86$
$16.645\,577\,3$ $3.395\,154\,843$ $1.0$ $5.843\,080\,683$ $60.453\,759\,86$
$16.645\,577\,3$ $3.395\,154\,843$ $1.0$ 1 DOF No regularizer
$156.125\,839\,4$ $99.883\,541\,01$ $99.333\,097\,76$ $98.181\,406\,73$
$2.580\,704\,274$ $156.077\,905\,5$ $99.927\,589\,34$ $99.401\,588\,47$
$98.108\,422\,33$ $2.579\,711\,796$ Whitening Nunes and Demiris (2021)
$156.178\,465\,3$ $99.952\,860\,01$ $99.514\,527\,98$ $98.261\,199\,02$
$2.582\,950\,586$ $156.817\,522\,3$ $99.879\,812\,64$ $99.383\,526\,44$
$98.329\,376\,88$ $2.580\,663\,18$ Divergence (Ours) $12.486\,649\,38$
$69.860\,514\,46$ $20.779\,218\,47$ $6.664\,941\,639$ $1.425\,272\,606$
$5.472\,864\,325$ $63.481\,962\,78$ $14.660\,480\,36$ $1.348\,473\,519$
$1.344\,124\,739$ Deformation (Ours) $9.007\,609\,466$ $68.956\,085\,51$
$18.861\,185\,38$ $4.773\,455\,209$ $1.402\,945\,911$ $5.793\,269\,017$
$64.022\,456\,64$ $16.108\,408\,44$ $2.751\,044\,423$ $1.361\,735\,211$ Div. +
Def. (Ours) $6.061\,261\,761$ $68.479\,370\,73$ $17.084\,346\,78$
$2.272\,174\,046$ $1.356\,512\,051$ $5.526\,156\,883$ $64.093\,066\,51$
$15.064\,542\,71$ $1.368\,051\,902$ $1.348\,987\,971$ 4 DOF Mitrokhin et al.
(2018) No regularizer $157.542\,172$ $99.968\,215\,14$ $99.642\,887\,96$
$98.666\,985\,69$ $2.637\,262\,171$ $157.335\,601\,7$ $99.942\,549\,34$
$99.525\,213\,17$ $98.444\,496\,33$ $2.618\,987\,077$ Whitening Nunes and
Demiris (2021) $157.732\,983\,9$ $99.969\,084\,4$ $99.661\,319\,66$
$98.713\,352\,43$ $2.604\,664\,639$ $156.118\,086\,8$ $99.912\,184\,62$
$99.258\,211\,52$ $97.929\,470\,05$ $2.609\,568\,805$ Divergence (Ours)
$14.345\,417\,85$ $90.835\,968\,08$ $41.618\,607\,43$ $10.816\,366\,73$
$1.349\,181\,226$ $10.428\,230\,07$ $91.377\,078\,59$ $41.628\,697\,72$
$9.433\,569\,862$ $1.214\,208\,323$ Deformation (Ours) $15.118\,467\,67$
$94.955\,666\,48$ $62.585\,701\,84$ $22.616\,035\,5$ $1.247\,289\,93$
$10.006\,316$ $90.146\,151\,44$ $39.451\,494\,29$ $8.670\,040\,138$
$1.249\,295\,937$ Div. + Def. (Ours) $10.059\,025\,13$ $90.646\,871\,98$
$40.607\,431\,78$ $8.583\,359\,441$ $1.263\,127\,072$ $10.393\,353\,83$
$91.018\,175\,47$ $41.812\,122\,68$ $9.396\,431\,802$ $1.226\,296\,549$

|  Original events |  IWE w/o regularizer |  IWE with regularizer |  Divergence map |  Deformation map
---|---|---|---|---|---
MVSEC Zhu et al. (2018) |  |  |  |  |
|  |  |  |  |
DSEC Gehrig et al. (2021) |  |  |  |  |
|  |  |  |  |
boxes_rot Mueggler et al. (2017) |  |  |  |  |
|  |  |  |  |
dynamic_rot Mueggler et al. (2017) |  |  |  |  |
|  |  |  |  |
|  (a) |  (b) |  (c) |  (d) |  (e)
Figure 6: _Proposed regularizers and collapse analysis_. The scene motion is
approximated by 1-DOF warp (zoom in/out) for MVSEC Zhu et al. (2018) and DSEC
Gehrig et al. (2021) sequences, and 3-DOF warp (rotation) for boxes and
dynamic ECD sequences Mueggler et al. (2017). (a) Original events. (b) Best
warp without regularization. Event collapse happens for 1-DOF warp. (c) Best
warp with regularization. (d) Divergence map ((10) is zero-based). (e)
Deformation map ((15), centered at 1). Our regularizers successfully penalize
event collapse and do not damage non-collapsing scenarios.

By contrast, our regularizers (Divergence and Deformation rows) work well to
mitigate the collapse, as observed in smaller AEE and $N$PE values. Compared
with the values of no regularizer or whitening Nunes and Demiris (2021), our
regularizers achieve more than 90% improvement for AEE on average. The AEE
values are high for optical flow standards ($4\mbox{--}8$ pix in MVSEC vs.
$0.5\mbox{--}1$ pixel Zhu et al. (2019), or $10\mbox{--}20$ pix in DSEC vs.
$2\mbox{--}5$ pix Gehrig et al. (2021)); however, this is due to the fact that
the warps used have very few DOFs ($\leq$4) compared to the considerably
higher DOFs ($2N_{p}$) of optical flow estimation algorithms. The same reason
explains the high 3PE values (standard in Geiger et al. (2013)): using an end-
point error threshold of 3 pix to consider that the flow is correctly
estimated does not convey the intended goal of inlier/outlier classification
for the low-DOF warps used. This is the reason why Tables 1 and 2 also report
10PE, 20PE metrics, and the values for the identity warp (zero flow). As
expected, for the range of AEE values in the tables, the 10PE and 20PE figures
demonstrate the large difference between methods suffering from collapse (20PE
$>$ 80%) and those that do not (20PE $<$ 1.1% for MVSEC and $<$22.6% for
DSEC).

The FWL values of our regularizers are moderately high ($\geq$1), indicating
that event alignment is better than that of the identity warp. However,
because the FWL depends on the number of events Stoffregen et al. (2020), it
is not easy to establish a global threshold to classify each method as
suffering from collapse or not. The AEE, 10PE, and 20PE are better for such a
classification.

Tables 1 and 2 also include the results of the use of both regularizers
simultaneously (“Div. + Def.”). The results improve across all sequences if
the data fidelity term is given by the variance loss, whereas they remain
approximately the same for the gradient magnitude loss. Regardless of the
choice of the proposed regularizer, the results in these tables clearly show
the effectiveness of our proposal, i.e., the large improvements compared with
prior works (rows “No regularizer” and Nunes and Demiris (2021)).

The collapse results are more visible in Figure 6, where we used the variance
loss. Without a regularizer, the events collapse in the MVSEC and DSEC
sequences. Our regularizers successfully mitigate overfitting, having a
remarkable impact on the estimated motion.

###  4.3 Effect of the Regularizers on Well-Posed Warps

Table 3 shows the results on the ECD dataset for a well-posed warp (3-DOF
rotational motion, in the benchmark). We use the variance loss and the Adam
optimizer Kingma and Ba (2015) with 100 iterations. All values in the table
(RMS error and FWL, with and without regularization, are very similar,
indicating that: (i) our regularizers do not affect the motion estimation
algorithm, and (ii) results without regularization are good due to the well-
posed warp. This is qualitatively shown in the bottom part of Figure 6. The
fluctuations of the divergence and deformation values away from those of the
identity warp ($0$ and $1$, respectively) are at least one order of magnitude
smaller than the collapse-enabled warps (e.g., $0.2$ vs. $2$).

Table 3: Results on ECD dataset Mueggler et al. (2017).

| boxes_rot | dynamic_rot
---|---|---
| RMS $\boldsymbol{\downarrow}$ | FWL $\boldsymbol{\uparrow}$ | RMS $\boldsymbol{\downarrow}$ | FWL $\boldsymbol{\uparrow}$
Ground truth pose | _ | $1.559\,016\,91$ | _ | $1.414\,303\,624$
No regularizer | $8.857\,855\,6$ | $1.561\,590\,243$ | $4.823\,395\,392$ | $1.420\,484\,652$
Divergence (Ours) | $9.236\,853\,563$ | $1.558\,060\,096$ | $4.826\,241\,768$ | $1.420\,484\,639$
Deformation (Ours) | $8.664\,275\,512$ | $1.560\,753\,596$ | $4.822\,492\,508$ | $1.420\,484\,1$

###  4.4 Sensitivity Analysis

The landscapes of loss functions as well as sensitivity analysis of $\lambda$
are shown in Figure 7, for the MVSEC experiments. Without regularizer
($\lambda=0$), all objective functions tested (variance, gradient magnitude,
and average timestamp Zhu et al. (2019)) suffer from event collapse, which is
the undesired global minimum of (20). Reaching the desired local optimum
depends on the optimizing algorithm and its initialization (e.g., starting
gradient descent close enough to the local optimum). Our regularizers
(divergence and deformation) change the landscape: the previously undesired
global minimum becomes local, and the desired minimum becomes the new global
one as $\lambda$ increases.

Specifically, the larger the weight $\lambda$, the smaller the effect of the
undesired minimum (at $h_{z}=1$). However, this is true only within some
reasonable range: a too large $\lambda$ discards the data-fidelity part $G$ in
(20), which is unwanted because it would remove the desired local optimum
(near $h_{z}\approx 0$). Minimizing (20) with only the regularizer is not
sensible.

Observe that for completeness, we include the average timestamp loss in the
last column. However, this loss also suffers from an undesired optimum in the
expansion region ($h_{z}\approx-1$). Our regularizers could be modified to
also remove this undesired optimum, but investigating this particular loss,
which was proposed as an alternative to the original contrast loss, is outside
the scope of this work.

|  |  |
---|---|---|---
No regularizer |  |  |
Divergence |  |  |
Deformation |  |  |
|  (a) |  (b) |  (c)
Figure 7: _Cost function landscapes over the warp parameter $h_{z}$ for_: (a)
Image variance Gallego et al. (2018), (b) gradient magnitude Gallego et al.
(2019), and (c) mean square of average timestamp Zhu et al. (2019). Data from
MVSEC Zhu et al. (2018) with dominant forward motion. The legend weights
denote $\lambda$ in (20).

###  4.5 Computational Complexity

Computing the regularizer(s) requires more computation than the non-
regularized objective. However, complexity is linear with the number of events
and the number of pixels, which is an advantage, and the warped events are
reutilized to compute the DIWE or IWA. Hence, the runtime is less than doubled
(warping is the dominant runtime term Gallego et al. (2019) and is computed
only once). The computational complexity of our regularized CMax framework is
$O(N_{e}+N_{p})$, the same as that of the non-regularized one.

###  4.6 Application to Motion Segmentation

Although most of the results on standard datasets comprise stationary scenes,
we have also provided results on a dynamic scene (from dataset Mueggler et al.
(2017)). Because the time spanned by each set of events processed is small,
the scene motion is also small (even for complicated objects like the person
in the bottom row of Figure 6), hence often a single warp fits the scene
reasonably well. In some scenarios, a single warp may not be enough to fit the
event data because there are distinctive motions in the scene of equal
importance. Our proposed regularizers can be extended to such more complex
scene motions. To this end, we demonstrate it with an example in Figure 8.

|  IWE with segmentation |  Divergence map
---|---|---
Without regularizer |  |
With regularizer (Ours) |  |
|  (a) |  (b)
Figure 8: _Application to Motion Segmentation_. (a) Output IWE, whose colors
(red and blue) represent different clusters of events (segmented according to
motion). (b) Divergence map. The range of divergence values is larger in the
presence of event collapse than in its absence. Our regularizer (divergence in
this example) mitigates the event collapse for this complex motion, even with
an independently moving object (IMO) in the scene.

Specifically, we use the MVSEC dataset, in a clip where the scene consists of
two motions: the ego-motion (forward motion of the recording vehicle) and the
motion of a car driving in the opposite direction in a nearby lane (an
independently moving object—IMO). We model the scene by using the combination
of two warps. Intuitively, the 1-DOF warp (6) describes the ego-motion, while
the feature flow (2 DOF) describes the IMO. Then, we apply the contrast
maximization approach (augmented with our regularizing terms) and the
expectation-maximization scheme in Stoffregen et al. (2019) to segment the
scene, to determine which events belong to each motion. The results in Figure
8 clearly show the effectiveness of our regularizer, even for such a
commonplace and complex scene. Without regularizers, (i) event collapse
appears in the ego-motion cluster of events and (ii) a considerable portion of
the events that correspond to ego-motion are assigned to the second cluster
(2-DOF warp), thus causing a segmentation failure. Our regularization approach
mitigates event collapse (bottom row of Figure 8) and provides the correct
segmentation: the 1-DOF warp fits the ego-motion and the feature flow (2-DOF
warp) fits the IMO.

##  5 Conclusions

We have analyzed the event collapse phenomenon of the CMax framework and
proposed collapse metrics using first principles of space-time deformation,
inspired by differential geometry and physics. Our experimental results on
publicly available datasets demonstrate that the proposed divergence and area-
based metrics mitigate the phenomenon for collapse-enabled warps and do not
harm well-posed warps. To the best of our knowledge, our regularizers are the
only effective solution compared to the unregularized CMax framework and
whitening. Our regularizers achieve, on average, more than 90% improvement on
optical flow endpoint error calculation (AEE) on collapse-enabled warps.

This is the first work that focuses on the paramount phenomenon of event
collapse. No prior work has analyzed this phenomenon in such detail or
proposed new regularizers without additional data or reparameterizing the
search space Zhu et al. (2019); Nunes and Demiris (2021); Peng et al. (2021).
As we analyzed various warps from 1 DOF to 4 DOFs, we hope that the ideas
presented here inspire further research to tackle more complex warp models.
Our work shows how the divergence and area-based deformation can be computed
for warps given by analytical formulas. For more complex warps, like those
used in dense optical flow estimation Zhu et al. (2019); Hagenaars et al.
(2021), the divergence or area-based deformation could be approximated by
using finite difference formulas.

yes

##  Appendix A Warp Models, Jacobians and Flow Divergence

###  A.1 Planar Motion — Euclidean Transformation on the Image Plane, $SE(2)$

If the point trajectories of an isometry are $\mathbf{x}(t)$, the warp is
given by Nunes and Demiris (2021)

$\begin{pmatrix}\mathbf{x}^{\prime}_{k}\\\
1\end{pmatrix}\sim\begin{pmatrix}\mathtt{R}(t_{k}\omega_{Z})&t_{k}\mathbf{v}\\\
\mathbf{0}^{\top}&1\end{pmatrix}^{-1}\begin{pmatrix}\mathbf{x}_{k}\\\
1\end{pmatrix},$ (21)

where $\mathbf{v},\omega_{Z}$ comprise the 3 DOFs of a translation and an in-
plane rotation. The in-plane rotation is

$\mathtt{R}(\phi)=\begin{pmatrix}\cos\phi&\;-\sin\phi\\\
\sin\phi&\cos\phi\end{pmatrix}.$ (22)

Since

$\begin{pmatrix}\mathtt{A}&\mathbf{b}\\\
\mathbf{0}^{\top}&1\end{pmatrix}^{-1}=\begin{pmatrix}\mathtt{A}^{-1}&-\mathtt{A}^{-1}\mathbf{b}\\\
\mathbf{0}^{\top}&1\end{pmatrix}$ (23)

and $\mathtt{R}^{-1}(\phi)=\mathtt{R}(-\phi)$, we have

$\begin{pmatrix}\mathbf{x}^{\prime}_{k}\\\
1\end{pmatrix}\sim\begin{pmatrix}\mathtt{R}(-t_{k}\omega_{Z})&-\mathtt{R}(-t_{k}\omega_{Z})(t_{k}\mathbf{v})\\\
\mathbf{0}^{\top}&1\end{pmatrix}\begin{pmatrix}\mathbf{x}_{k}\\\
1\end{pmatrix}.$ (24)

Hence, in Euclidean coordinates the warp is

$\mathbf{x}^{\prime}_{k}=\mathtt{R}(-t_{k}\omega_{Z})(\mathbf{x}_{k}-t_{k}\mathbf{v}).$
(25)

The Jacobian and its determinant are:

$\mathtt{J}_{k}=\frac{\partial\mathbf{x}^{\prime}_{k}}{\partial\mathbf{x}_{k}}=\mathtt{R}(-t_{k}\omega_{Z}),$
(26) $\det(\mathtt{J}_{k})=1.$ (27)

The flow corresponding to (25) is:

$\mathbf{f}=\frac{\partial\mathbf{x}^{\prime}}{\partial
t}=\mathtt{R}^{\top}\Bigl{(}\frac{\pi}{2}+t\omega_{Z}\Bigr{)}(\mathbf{x}-t\mathbf{v})\omega_{Z}-\mathtt{R}(-t\omega_{Z})\mathbf{v},$
(28)

whose divergence is

$\nabla\cdot\mathbf{f}=-2\omega_{Z}\sin(t\omega_{Z}).$ (29)

Hence, for small angles $|t\omega_{Z}|\ll 1$, the divergence of the flow
vanishes.

In short, this warp has the same determinant and approximate zero divergence
as the 2-DOF feature flow warp (Section 3.5.1), which is well-behaved. Note,
however, that the trajectories are not straight in space-time.

###  A.2 3-DOF Camera Rotation, $SO(3)$

Using calibrated and homogeneous coordinates, the warp is given by Gallego and
Scaramuzza (2017); Gallego et al. (2018)

$\mathbf{x}_{k}^{h\prime}\sim\mathtt{R}(t_{k}\boldsymbol{\omega})\,\mathbf{x}^{h}_{k},$
(30)

where
$\boldsymbol{\theta}=\boldsymbol{\omega}=(\omega_{1},\omega_{2},\omega_{3})^{\top}$
is the angular velocity, and $\mathtt{R}$ ($3\times 3$ rotation matrix in
space) is parametrized using exponential coordinates (Rodrigues rotation
formula Murray et al. (1994); Gallego and Yezzi (2014)).

By the chain rule, the Jacobian is:

$\mathtt{J}_{k}=\frac{\partial\mathbf{x}^{\prime}_{k}}{\partial\mathbf{x}_{k}}=\frac{\partial\mathbf{x}^{\prime}_{k}}{\partial\mathbf{x}^{h\prime}_{k}}\frac{\partial\mathbf{x}^{h\prime}_{k}}{\partial\mathbf{x}^{h}_{k}}\frac{\partial\mathbf{x}^{h}_{k}}{\partial\mathbf{x}_{k}}=\frac{1}{(\mathbf{x}^{h\prime}_{k})_{3}}\begin{pmatrix}1&0&-x^{\prime}_{k}\\\
0&1&-y^{\prime}_{k}\end{pmatrix}\mathtt{R}(t_{k}\boldsymbol{\omega})\begin{pmatrix}1&0\\\
0&1\\\ 0&0\end{pmatrix}.$ (31)

Letting $\mathbf{r}_{3,k}^{\top}$ be the third row of
$\mathtt{R}(t_{k}\boldsymbol{\omega})$, and using (32)–(34) in Gallego et al.
(2011), gives

$\det(\mathtt{J}_{k})=(\mathbf{r}_{3,k}^{\top}\mathbf{x}^{h}_{k})^{-3}.$ (32)

#### Connection between divergence and deformation maps

If the rotation angle $t_{k}\|\boldsymbol{\omega}\|$ is small, using the first
two terms of the exponential map we approximate
$\mathtt{R}(t_{k}\boldsymbol{\omega})\approx\mathtt{Id}+(t_{k}\boldsymbol{\omega})^{\wedge}$,
where the hat operator ∧ in $SO(3)$ represents the cross product matrix
Barfoot (2015). Then,
$\mathbf{r}_{3,k}^{\top}\mathbf{x}^{h}_{k}\approx(-t_{k}\omega_{2},t_{k}\omega_{1},1)^{\top}(x_{k},y_{k},1)=1+(y_{k}\omega_{1}-x_{k}\omega_{2})t_{k}$.
Substituting this expression into (32) and using the first two terms in
Taylor’s expansion around $z=0$ of $(1+z)^{-3}\approx 1-3z+6z^{2}$ (convergent
for $|z|<1$) gives $\det(\mathtt{J}_{k})\approx
1+3(x_{k}\omega_{2}-y_{k}\omega_{1})t_{k}$. Notably, the divergence (18) and
the approximate amplification factor depend linearly on
$3(x_{k}\omega_{2}-y_{k}\omega_{1})$. This resemblance is seen in the
divergence and deformation maps of the bottom rows in Figure 6 (ECD dataset).

###  A.3 4-DOF In-Plane Camera Motion Approximation

The warp presented in Mitrokhin et al. (2018),

$\mathbf{x}^{\prime}_{k}=\mathbf{x}_{k}-t_{k}\,\bigl{(}\mathbf{v}+(h_{z}+1)\mathtt{R}(\phi)\mathbf{x}_{k}-\mathbf{x}_{k}\bigr{)}$
(33)

has 4 DOFs: $\boldsymbol{\theta}=(\mathbf{v},\phi,h_{z})^{\top}$. The Jacobian
and its determinant are:

$\mathtt{J}_{k}=\frac{\partial\mathbf{x}^{\prime}_{k}}{\partial\mathbf{x}_{k}}=(1+t_{k})\mathtt{Id}-(h_{z}+1)t_{k}\mathtt{R}(\phi),$
(34)
$\det(\mathtt{J}_{k})=(1+t_{k})^{2}-2(1+t_{k})t_{k}(h_{z}+1)\cos\phi+t_{k}^{2}(h_{z}+1)^{2}.$
(35)

The flow corresponding to (33) is given by

$\mathbf{f}=\frac{\partial\mathbf{x}^{\prime}}{\partial
t}=-\bigl{(}\mathbf{v}+(h_{z}+1)\mathtt{R}(\phi)\mathbf{x}-\mathbf{x}\bigr{)},$
(36)

whose divergence is:

$\displaystyle\nabla\cdot\mathbf{f}$
$\displaystyle=-(h_{z}+1)\nabla\cdot\bigl{(}\mathtt{R}(\phi)\mathbf{x}\bigr{)}+\nabla\cdot\mathbf{x}$
(37) $\displaystyle=2-2(h_{z}+1)\cos(\phi).$ (38)

As particular cases of this warp, one can identify:

  * •

1-DOF Zoom in/out ($\mathbf{v}=\mathbf{0},\phi=0$).
$\mathbf{x}^{\prime}_{k}=(1-t_{k}h_{z})\mathbf{x}_{k}$.

  * •

2-DOF translation ($\phi=0,h_{z}=0$).
$\mathbf{x}^{\prime}_{k}=\mathbf{x}_{k}-t_{k}\mathbf{v}$.

  * •

1-DOF “rotation” ($\mathbf{v}=\mathbf{0},h_{z}=0$).
$\mathbf{x}^{\prime}_{k}=\mathbf{x}_{k}-t_{k}\,\bigl{(}\mathtt{R}(\phi)\mathbf{x}_{k}-\mathbf{x}_{k}\bigr{)}$.
Using a couple of approximations of the exponential map in $SO(2)$, we obtain

$\displaystyle\mathbf{x}^{\prime}_{k}$
$\displaystyle=\mathbf{x}_{k}-t_{k}\,\bigl{(}\mathtt{R}(\phi)-\mathtt{Id}\bigr{)}\mathbf{x}_{k}$
(39) $\displaystyle\approx\mathbf{x}_{k}-t_{k}\phi^{\wedge}\mathbf{x}_{k}$
$\displaystyle\text{if }\phi\text{ is small}$ (40)
$\displaystyle=(\mathtt{Id}+(-t_{k}\phi)^{\wedge})\mathbf{x}_{k}$ (41)
$\displaystyle\approx\mathtt{R}(-t_{k}\phi)\mathbf{x}_{k}$
$\displaystyle\text{if }t_{k}\phi\text{ is small}.$ (42)

Hence, $\phi$ plays the role of a small angular velocity $\omega_{Z}$ around
the camera’s optical axis $Z$, i.e., in-plane rotation.

  * •

3-DOF planar motion (“isometry”) ($h_{z}=0$). Using the previous result, the
warp splits into translational and rotational components:

$\displaystyle\mathbf{x}^{\prime}_{k}$
$\displaystyle=\mathbf{x}_{k}-t_{k}\,\bigl{(}\mathbf{v}+\mathtt{R}(\phi)\mathbf{x}_{k}-\mathbf{x}_{k}\bigr{)}$
(43)
$\displaystyle\stackrel{{\scriptstyle\eqref{eq:maryland:approxrot}}}{{\approx}}-t_{k}\mathbf{v}+\mathtt{R}(-t_{k}\phi)\mathbf{x}_{k}.$
(44)

###  A.4 4-DOF Similarity Transformation on the Image Plane, Sim(2)

Another 4-DOF warp is proposed in Nunes and Demiris (2021). Its DOFs are the
linear, angular and scaling velocities on the image plane:
$\boldsymbol{\theta}=(\mathbf{v},\omega_{Z},s)^{\top}$.

Letting $\beta_{k}=1+t_{k}s$, the warp is:

$\begin{pmatrix}\mathbf{x}^{\prime}_{k}\\\
1\end{pmatrix}\sim\begin{pmatrix}\beta_{k}\mathtt{R}(t_{k}\omega_{Z})&t_{k}\mathbf{v}\\\
\mathbf{0}^{\top}&1\end{pmatrix}^{-1}\begin{pmatrix}\mathbf{x}_{k}\\\
1\end{pmatrix}.$ (45)

Using (23) gives

$\begin{pmatrix}\mathbf{x}^{\prime}_{k}\\\
1\end{pmatrix}\sim\begin{pmatrix}\beta^{-1}_{k}\mathtt{R}(-t_{k}\omega_{Z})&-\beta^{-1}_{k}\mathtt{R}(-t_{k}\omega_{Z})(t_{k}\mathbf{v})\\\
\mathbf{0}^{\top}&1\end{pmatrix}\begin{pmatrix}\mathbf{x}_{k}\\\
1\end{pmatrix}.$ (46)

Hence, in Euclidean coordinates the warp is

$\mathbf{x}^{\prime}_{k}=\beta^{-1}_{k}\mathtt{R}(-t_{k}\omega_{Z})(\mathbf{x}_{k}-t_{k}\mathbf{v}).$
(47)

The Jacobian and its determinant are:

$\mathtt{J}_{k}=\frac{\partial\mathbf{x}^{\prime}_{k}}{\partial\mathbf{x}_{k}}=\beta^{-1}_{k}\mathtt{R}(-t_{k}\omega_{Z}),$
(48) $\det(\mathtt{J}_{k})=\beta^{-2}_{k}=\frac{1}{(1+t_{k}s)^{2}}.$ (49)

The following result will be useful to simplify equations. For a 2D rotation
$\mathtt{R}(\phi(t))$, it holds that:

$\displaystyle\frac{\partial\mathtt{R}(\phi(t))}{\partial
t}=-\mathtt{R}^{\top}\Bigl{(}\frac{\pi}{2}-\phi\Bigr{)}\,\frac{\partial\phi}{\partial
t}.$ (50)

To compute the flow of (47), there are three time-dependent factors. Hence,
applying the product rule we obtain three terms, and substituting (50) (with
$\phi=-t\omega_{Z}$) gives:

$\displaystyle\mathbf{f}_{k}=\Bigl{(}\frac{\partial\beta^{-1}_{k}}{\partial
t_{k}}\mathtt{R}(-t_{k}\omega_{Z})+\beta^{-1}_{k}\omega_{Z}\mathtt{R}^{\top}\bigl{(}\frac{\pi}{2}+t_{k}\omega_{Z}\bigr{)}\Bigr{)}(\mathbf{x}_{k}-t_{k}\mathbf{v})-\beta^{-1}_{k}\mathtt{R}(-t_{k}\omega_{Z})\mathbf{v},$
(51)

where, by the chain rule,

$\displaystyle\frac{\partial\beta^{-1}_{k}}{\partial
t_{k}}=-\beta^{-2}_{k}\frac{\partial\beta_{k}}{\partial
t_{k}}=-\beta^{-2}_{k}s=-\frac{s}{(1+t_{k}s)^{2}}.$ (52)

Hence, the divergence of the flow is:

$\displaystyle\nabla\cdot\mathbf{f}_{k}$
$\displaystyle=\frac{\partial\beta^{-1}_{k}}{\partial
t_{k}}\nabla\cdot\Bigl{(}\mathtt{R}(-t_{k}\omega_{Z})\mathbf{x}_{k}\Bigr{)}+\beta^{-1}_{k}\omega_{Z}\nabla\cdot\Bigl{(}\mathtt{R}^{\top}\bigl{(}\frac{\pi}{2}+t_{k}\omega_{Z}\bigr{)}\mathbf{x}_{k}\Bigr{)}$
(53) $\displaystyle=\frac{\partial\beta^{-1}_{k}}{\partial
t_{k}}2\cos(t_{k}\omega_{Z})+\beta^{-1}_{k}\omega_{Z}2\sin(-t_{k}\omega_{Z})$
(54)

The formulas for $SE(2)$ are obtained from the above ones with $s=0$ (i.e.,
$\beta_{k}=1$).

References

## References

  * Delbruck (2008) Delbruck, T.  Frame-free dynamic digital vision.  In Proceedings of the Proc. Int. Symp. Secure-Life Electron., 2008, pp. 21–26.  https://doi.org/10.5167/uzh-17620.
  * Suh et al. (2020) Suh, Y.; Choi, S.; Ito, M.; Kim, J.; Lee, Y.; Seo, J.; Jung, H.; Yeo, D.H.; Namgung, S.; Bong, J.; et al.  A 1280x960 Dynamic Vision Sensor with a 4.95-$\mu$m Pixel Pitch and Motion Artifact Minimization.  In Proceedings of the IEEE Int. Symp. Circuits Syst. (ISCAS), 2020\.  https://doi.org/10.1109/ISCAS45731.2020.9180436.
  * Finateu et al. (2020) Finateu, T.; Niwa, A.; Matolin, D.; Tsuchimoto, K.; Mascheroni, A.; Reynaud, E.; Mostafalu, P.; Brady, F.; Chotard, L.; LeGoff, F.; et al.  A 1280x720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86$\mu$m Pixels, 1.066GEPS Readout, Programmable Event-Rate Controller and Compressive Data-Formatting Pipeline.  In Proceedings of the IEEE Intl. Solid-State Circuits Conf. (ISSCC), 2020, pp. 112–114.  https://doi.org/10.1109/ISSCC19947.2020.9063149.
  * Gallego et al. (2020) Gallego, G.; Delbruck, T.; Orchard, G.; Bartolozzi, C.; Taba, B.; Censi, A.; Leutenegger, S.; Davison, A.; Conradt, J.; Daniilidis, K.; et al.  Event-based Vision: A Survey.  IEEE Trans. Pattern Anal. Mach. Intell. 2020.  https://doi.org/10.1109/TPAMI.2020.3008413.
  * Gallego and Scaramuzza (2017) Gallego, G.; Scaramuzza, D.  Accurate Angular Velocity Estimation with an Event Camera.  IEEE Robot. Autom. Lett. 2017, 2, 632–639.  https://doi.org/10.1109/LRA.2016.2647639.
  * Kim and Kim (2021) Kim, H.; Kim, H.J.  Real-Time Rotational Motion Estimation With Contrast Maximization Over Globally Aligned Events.  IEEE Robot. Autom. Lett. 2021, 6, 6016–6023.  https://doi.org/10.1109/LRA.2021.3088793.
  * Zhu et al. (2017a) Zhu, A.Z.; Atanasov, N.; Daniilidis, K.  Event-Based Feature Tracking with Probabilistic Data Association.  In Proceedings of the IEEE Int. Conf. Robot. Autom. (ICRA), 2017, pp. 4465–4470.  https://doi.org/10.1109/ICRA.2017.7989517.
  * Zhu et al. (2017b) Zhu, A.Z.; Atanasov, N.; Daniilidis, K.  Event-based Visual Inertial Odometry.  In Proceedings of the IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2017, pp. 5816–5824.  https://doi.org/10.1109/CVPR.2017.616.
  * Seok and Lim (2020) Seok, H.; Lim, J.  Robust Feature Tracking in DVS Event Stream using Bezier Mapping.  In Proceedings of the IEEE Winter Conf. Appl. Comput. Vis. (WACV), 2020, pp. 1647–1656.  https://doi.org/10.1109/WACV45572.2020.9093607.
  * Stoffregen and Kleeman (2019) Stoffregen, T.; Kleeman, L.  Event Cameras, Contrast Maximization and Reward Functions: an Analysis.  In Proceedings of the IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2019, pp. 12292–12300.  https://doi.org/10.1109/CVPR.2019.01258.
  * Dardelet et al. (2021) Dardelet, L.; Benosman, R.; Ieng, S.H.  An Event-by-Event Feature Detection and Tracking Invariant to Motion Direction and Velocity.  TechRxiv preprint 2021.  https://doi.org/10.36227/techrxiv.17013824.v1.
  * Gallego et al. (2018) Gallego, G.; Rebecq, H.; Scaramuzza, D.  A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation.  In Proceedings of the IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2018, pp. 3867–3876.  https://doi.org/10.1109/CVPR.2018.00407.
  * Gallego et al. (2019) Gallego, G.; Gehrig, M.; Scaramuzza, D.  Focus Is All You Need: Loss Functions For Event-based Vision.  In Proceedings of the IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2019, pp. 12272–12281.  https://doi.org/10.1109/CVPR.2019.01256.
  * Peng et al. (2021) Peng, X.; Gao, L.; Wang, Y.; Kneip, L.  Globally-Optimal Contrast Maximisation for Event Cameras.  IEEE Trans. Pattern Anal. Mach. Intell. 2021, pp. 1–1.  https://doi.org/10.1109/TPAMI.2021.3053243.
  * Rebecq et al. (2018) Rebecq, H.; Gallego, G.; Mueggler, E.; Scaramuzza, D.  EMVS: Event-based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time.  Int. J. Comput. Vis. 2018, 126, 1394–1414.  https://doi.org/10.1007/s11263-017-1050-6.
  * Zhu et al. (2019) Zhu, A.Z.; Yuan, L.; Chaney, K.; Daniilidis, K.  Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion.  In Proceedings of the IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2019, pp. 989–997.  https://doi.org/10.1109/CVPR.2019.00108.
  * Paredes-Valles et al. (2019) Paredes-Valles, F.; Scheper, K.Y.W.; de Croon, G.C.H.E.  Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception.  IEEE Trans. Pattern Anal. Mach. Intell. 2019.  https://doi.org/10.1109/TPAMI.2019.2903179.
  * Hagenaars et al. (2021) Hagenaars, J.J.; Paredes-Valles, F.; de Croon, G.C.H.E.  Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2021, Vol. 34, pp. 7167–7179.
  * Shiba et al. (2022) Shiba, S.; Aoki, Y.; Gallego, G.  Secrets of Event-based Optical Flow.  In Proceedings of the Eur. Conf. Comput. Vis. (ECCV), 2022.
  * Mitrokhin et al. (2018) Mitrokhin, A.; Fermuller, C.; Parameshwara, C.; Aloimonos, Y.  Event-based Moving Object Detection and Tracking.  In Proceedings of the IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS), 2018, pp. 1–9.  https://doi.org/10.1109/IROS.2018.8593805.
  * Stoffregen et al. (2019) Stoffregen, T.; Gallego, G.; Drummond, T.; Kleeman, L.; Scaramuzza, D.  Event-Based Motion Segmentation by Motion Compensation.  In Proceedings of the Int. Conf. Comput. Vis. (ICCV), 2019, pp. 7243–7252.  https://doi.org/10.1109/ICCV.2019.00734.
  * Zhou et al. (2021) Zhou, Y.; Gallego, G.; Lu, X.; Liu, S.; Shen, S.  Event-based Motion Segmentation with Spatio-Temporal Graph Cuts.  IEEE Trans. Neural Netw. Learn. Syst. 2021, pp. 1–13.  https://doi.org/10.1109/TNNLS.2021.3124580.
  * Parameshwara et al. (2021) Parameshwara, C.M.; Sanket, N.J.; Singh, C.D.; Fermüller, C.; Aloimonos, Y.  0-MMS: Zero-shot multi-motion segmentation with a monocular event camera.  In Proceedings of the IEEE Int. Conf. Robot. Autom. (ICRA), 2021.  https://doi.org/10.1109/ICRA48506.2021.9561755.
  * Lu et al. (2021) Lu, X.; Zhou, Y.; Shen, S.  Event-based Motion Segmentation by Cascaded Two-Level Multi-Model Fitting.  In Proceedings of the IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS), 2021, pp. 4445–4452.  https://doi.org/10.1109/IROS51168.2021.9636307.
  * Duan et al. (2021) Duan, P.; Wang, Z.; Shi, B.; Cossairt, O.; Huang, T.; Katsaggelos, A.  Guided Event Filtering: Synergy between Intensity Images and Neuromorphic Events for High Performance Imaging.  IEEE Trans. Pattern Anal. Mach. Intell. 2021, pp. 1–1.  https://doi.org/10.1109/TPAMI.2021.3113344.
  * Zhang et al. (2021) Zhang, Z.; Yezzi, A.; Gallego, G.  Image Reconstruction from Events. Why learn it?  arXiv e-prints 2021.
  * Nunes and Demiris (2021) Nunes, U.M.; Demiris, Y.  Robust Event-based Vision Model Estimation by Dispersion Minimisation.  IEEE Trans. Pattern Anal. Mach. Intell. 2021.  https://doi.org/10.1109/TPAMI.2021.3130049.
  * Gu et al. (2021) Gu, C.; Learned-Miller, E.; Sheldon, D.; Gallego, G.; Bideau, P.  The Spatio-Temporal Poisson Point Process: A Simple Model for the Alignment of Event Camera Data.  In Proceedings of the Int. Conf. Comput. Vis. (ICCV), 2021, pp. 13495–13504.  https://doi.org/10.1109/ICCV48922.2021.01324.
  * Liu et al. (2020) Liu, D.; Parra, A.; Chin, T.J.  Globally Optimal Contrast Maximisation for Event-Based Motion Estimation.  In Proceedings of the IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2020, pp. 6348–6357.  https://doi.org/10.1109/CVPR42600.2020.00638.
  * Stoffregen and Kleeman (2017) Stoffregen, T.; Kleeman, L.  Simultaneous Optical Flow and Segmentation (SOFAS) using Dynamic Vision Sensor.  In Proceedings of the Australasian Conf. Robot. Autom. (ACRA), 2017.
  * Ozawa et al. (2022) Ozawa, T.; Sekikawa, Y.; Saito, H.  Accuracy and Speed Improvement of Event Camera Motion Estimation Using a Bird’s-Eye View Transformation.  Sensors 2022, 22.  https://doi.org/10.3390/s22030773.
  * Lichtsteiner et al. (2008) Lichtsteiner, P.; Posch, C.; Delbruck, T.  A 128$\times$128 120 dB 15 $\mu$s latency asynchronous temporal contrast vision sensor.  IEEE J. Solid-State Circuits 2008, 43, 566–576.  https://doi.org/10.1109/JSSC.2007.914337.
  * Ng et al. (2022) Ng, M.; Er, Z.M.; Soh, G.S.; Foong, S.  Aggregation Functions For Simultaneous Attitude And Image Estimation With Event Cameras At High Angular Rates.  IEEE Robot. Autom. Lett. 2022, pp. 1–1.  https://doi.org/10.1109/LRA.2022.3148982.
  * Zhu et al. (2018) Zhu, A.Z.; Thakur, D.; Ozaslan, T.; Pfrommer, B.; Kumar, V.; Daniilidis, K.  The Multivehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception.  IEEE Robot. Autom. Lett. 2018, 3, 2032–2039.  https://doi.org/10.1109/lra.2018.2800793.
  * Murray et al. (1994) Murray, R.M.; Li, Z.; Sastry, S.  A Mathematical Introduction to Robotic Manipulation; CRC Press, 1994\.
  * Gallego and Yezzi (2014) Gallego, G.; Yezzi, A.  A Compact Formula for the Derivative of a 3-D Rotation in Exponential Coordinates.  J. Math. Imaging Vis. 2014, 51, 378–384.  https://doi.org/10.1007/s10851-014-0528-x.
  * Corke (2017) Corke, P.  Robotics, Vision and Control: Fundamental Algorithms in MATLAB; Springer Tracts in Advanced Robotics, Springer, 2017.  https://doi.org/10.1007/978-3-319-54413-7.
  * Gallego et al. (2011) Gallego, G.; Yezzi, A.; Fedele, F.; Benetazzo, A.  A Variational Stereo Method for the Three-Dimensional Reconstruction of Ocean Waves.  IEEE Trans. Geosci. Remote Sens. 2011, 49, 4445–4457.  https://doi.org/10.1109/TGRS.2011.2150230.
  * Gehrig et al. (2021) Gehrig, M.; Aarents, W.; Gehrig, D.; Scaramuzza, D.  DSEC: A Stereo Event Camera Dataset for Driving Scenarios.  IEEE Robot. Autom. Lett. 2021.  https://doi.org/10.1109/LRA.2021.3068942.
  * Mueggler et al. (2017) Mueggler, E.; Rebecq, H.; Gallego, G.; Delbruck, T.; Scaramuzza, D.  The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM.  Int. J. Robot. Research 2017, 36, 142–149.  https://doi.org/10.1177/0278364917691115.
  * Gehrig et al. (2021) Gehrig, M.; Millhäusler, M.; Gehrig, D.; Scaramuzza, D.  E-RAFT: Dense Optical Flow from Event Cameras.  In Proceedings of the Int. Conf. 3D Vision (3DV), 2021.  https://doi.org/10.1109/3DV53792.2021.00030.
  * Nagata et al. (2021) Nagata, J.; Sekikawa, Y.; Aoki, Y.  Optical Flow Estimation by Matching Time Surface with Event-Based Cameras.  Sensors 2021, 21.  https://doi.org/10.3390/s21041150.
  * Taverni et al. (2018) Taverni, G.; Moeys, D.P.; Li, C.; Cavaco, C.; Motsnyi, V.; Bello, D.S.S.; Delbruck, T.  Front and Back Illuminated Dynamic and Active Pixel Vision Sensors Comparison.  IEEE Trans. Circuits Syst. II 2018, 65, 677–681.  https://doi.org/10.1109/TCSII.2018.2824899.
  * Zhu et al. (2018) Zhu, A.Z.; Yuan, L.; Chaney, K.; Daniilidis, K.  EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras.  In Proceedings of the Robotics: Science and Systems (RSS), 2018.  https://doi.org/10.15607/RSS.2018.XIV.062.
  * Rosinol Vidal et al. (2018) Rosinol Vidal, A.; Rebecq, H.; Horstschaefer, T.; Scaramuzza, D.  Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios.  IEEE Robot. Autom. Lett. 2018, 3, 994–1001.  https://doi.org/10.1109/LRA.2018.2793357.
  * Rebecq et al. (2017) Rebecq, H.; Horstschäfer, T.; Gallego, G.; Scaramuzza, D.  EVO: A Geometric Approach to Event-based 6-DOF Parallel Tracking and Mapping in Real-Time.  IEEE Robot. Autom. Lett. 2017, 2, 593–600.  https://doi.org/10.1109/LRA.2016.2645143.
  * Mueggler et al. (2018) Mueggler, E.; Gallego, G.; Rebecq, H.; Scaramuzza, D.  Continuous-Time Visual-Inertial Odometry for Event Cameras.  IEEE Trans. Robot. 2018, 34, 1425–1440.  https://doi.org/10.1109/tro.2018.2858287.
  * Zhou et al. (2021) Zhou, Y.; Gallego, G.; Shen, S.  Event-based Stereo Visual Odometry.  IEEE Trans. Robot. 2021, 37, 1433–1450.  https://doi.org/10.1109/TRO.2021.3062252.
  * Brandli et al. (2014) Brandli, C.; Berner, R.; Yang, M.; Liu, S.C.; Delbruck, T.  A 240x180 130dB 3$\mu$s Latency Global Shutter Spatiotemporal Vision Sensor.  IEEE J. Solid-State Circuits 2014, 49, 2333–2341.  https://doi.org/10.1109/JSSC.2014.2342715.
  * Stoffregen et al. (2020) Stoffregen, T.; Scheerlinck, C.; Scaramuzza, D.; Drummond, T.; Barnes, N.; Kleeman, L.; Mahony, R.  Reducing the Sim-to-Real Gap for Event Cameras.  In Proceedings of the Eur. Conf. Comput. Vis. (ECCV), 2020.
  * Bergstra et al. (2011) Bergstra, J.; Bardenet, R.; Bengio, Y.; Kégl, B.  Algorithms for Hyper-Parameter Optimization.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2011, Vol. 24.
  * Geiger et al. (2013) Geiger, A.; Lenz, P.; Stiller, C.; Urtasun, R.  Vision meets robotics: The KITTI dataset.  Int. J. Robot. Research 2013, 32, 1231–1237.  https://doi.org/10.1177/0278364913491297.
  * Kingma and Ba (2015) Kingma, D.P.; Ba, J.L.  Adam: A Method for Stochastic Optimization.  Int. Conf. Learn. Representations (ICLR) 2015.
  * Barfoot (2015) Barfoot, T.D.  State Estimation for Robotics - A Matrix Lie Group Approach; Cambridge University Press, 2015.