11(2.5, 0.45)

This journal paper has been accepted for publication at Advanced Intelligent
Systems, 2022.

# A Fast Geometric Regularizer to Mitigate Event Collapse in the Contrast
Maximization Framework

Shintaro Shiba1,2, Yoshimitsu Aoki1, Guillermo Gallego2,3 1 Department of
Electronics and Electrical Engineering, Faculty of Science and Technology,
Keio University, Kanagawa, Japan. 2 Department of Electrical Engineering and
Computer Science, Technische Universität Berlin, Berlin, Germany. 3 Einstein
Center Digital Future and Science of Intelligence Excellence Cluster, Berlin,
Germany. Preprint of paper accepted at Advanced Intelligent Systems, 2022.
doi: 10.1002/aisy.202200251. 0000-0001-6053-2285 0000-0002-2672-9241

###### Abstract

Event cameras are emerging vision sensors and their advantages are suitable
for various applications such as autonomous robots. Contrast maximization
(CMax), which provides state-of-the-art accuracy on motion estimation using
events, may suffer from an overfitting problem called event collapse. Prior
works are computationally expensive or cannot alleviate the overfitting, which
undermines the benefits of the CMax framework. We propose a novel,
computationally efficient regularizer based on geometric principles to
mitigate event collapse. The experiments show that the proposed regularizer
achieves state-of-the-art accuracy results, while its reduced computational
complexity makes it two to four times faster than previous approaches. To the
best of our knowledge, our regularizer is the only effective solution for
event collapse without trading off runtime. We hope our work opens the door
for future applications that unlocks the advantages of event cameras.

## Multimedia Material

Project page: https://github.com/tub-rip/event_collapse

##  1 Introduction

Event cameras [1, 2] are an emerging technology attracting considerable
attention because of their advantages, such as low latency, high dynamic range
and data/power efficiency. These advantages are especially suitable for
applications that require low-latency vision-based processing, such as
autonomous driving [3, 4]. The neuromorphic principle of operation of event
cameras makes them naturally respond to the scene dynamics. Therefore, they
have been used in motion estimation tasks [5, 6], such as Simultaneous
Localization and Mapping [7, 8, 9, 10, 11, 12], optical flow estimation [13,
14, 15, 16], particle image velocimetry [17], motion segmentation [18, 19],
obstacle avoidance [20], and space situational awareness [21].

Figure 1: _Sample application of event cameras._ Top: The advantages of event
cameras are beneficial for robotics applications, such as autonomous driving.
Bottom: The proposed regularizer discourages event collapse (left), and
reveals sharp edges of the scene in a computationally efficient manner
(right). (Top image licensed from Stock Photo ID 1102269152).

Contrast maximization (CMax) [22] is a powerful event processing framework
that achieves state-of-the-art accuracy in various motion estimation tasks. On
the other hand, it may suffer from an overfitting problem called _event
collapse_ , where the optimizer converges to an undesired global optimum [23].
Prior works have tackled event collapse in several ways, such as whitening the
data [24], reformulating the task (e.g., by providing additional depth
information [24]), or adding a regularizer to improve the optimization
landscape [23]. However, these proposals present shortcomings: ($i$) assuming
known depth data is task-specific and requires an additional sensor such as a
LiDAR or a stereo setup, and ($ii$) the above regularizing techniques may not
be effective [24] or require considerable extra computation [23]. Towards more
practical application of event cameras, it is paramount to effectively
alleviate event collapse in a computationally-efficient manner.

Figure 2: _Method overview_. The proposed regularizer (blue line) is based on
geometric principles and solely relies on motion parameters
$\boldsymbol{\theta}$, while previous approaches (dashed line) are built from
warped events [23]. Adapted with permission from Ref. [25], 2019, Gallego et
al.

This paper proposes a novel, computationally efficient regularizer to mitigate
event collapse in the CMax framework (Figure 1). From a theoretical point of
view, the regularizer is designed based on geometric principles of motion
field deformation (measuring area rate of change along point trajectories). In
contrast to previous methods, the regularizer does not depend on the event
data; it only depends on the motion hypothesis (i.e., the warp). This is
desirable because events may not be equally distributed on the space-time
image domain, and motion hypotheses provide information even in homogeneous
brightness regions (where events are scarce). From a practical point of view,
the proposed regularizer drastically reduces computational complexity, being
two to four times faster than previous solutions, while achieving state-of-
the-art results to mitigate event collapse.

The above contributions open the door to efficient and interpretable
regularizers for motion estimation problems with geometrically-meaningful
parametrizations.

##  2 Related Work

Motion estimation from event data is an incipient research field. Event
cameras respond to moving edges in the scene, and edges carry the essential
information for motion estimation. Hence event cameras have numerous
applications in motion-related tasks (ego-motion and/or scene-motion). We
refer to [26] for a comprehensive review. State-of-the-art approaches are
based on the concept of _event alignment_ or motion compensation [22, 25, 24,
27], estimating the motion parameters that associate events corresponding to
the same scene edge. In particular, Contrast Maximization (CMax) [22] seeks
the motion parameters that maximize the contrast (or focus score) of an image
of warped events (IWE). It has been used on various motion estimation tasks,
such as rotational motion [28, 29, 27], feature flow [30, 9, 31, 32],
homographic motion [22, 24, 33], optical flow [14, 15, 16], motion
segmentation [34, 18, 19, 35] and 3D reconstruction [12, 36]. However, the
optimizer may converge to an undesired global optimum that pushes events into
too few pixels (i.e., event collapse).

Event collapse is originally analyzed in detail by [23]. It may occur or not
depending on the task, the space of motion hypotheses and the data. Hence,
previous works for tackling event collapse can be categorized based on tasks.
For example, optical flow estimation has a high number of degrees-of-freedom
(DOFs) ($2N_{p}$, where $N_{p}$ is the number of pixels), i.e., motion
parameters, and is a collapse-enabled problem. A common approach to mitigate
collapse for this task is to add a strong regularizer, such as the classical
Charbonnier loss [37], to encourage smoothness of the flow [14, 32]. Another
approach consists of increasing the well-posedness of the problem by using a
tile-based motion field and a multi-reference focus loss [16]. However, event
collapse may still appear in some parts of the image plane.

Ego-motion estimation problems, which are the main focus of this work,
parametrize motion over the image space with relatively lower DOFs. The well-
posedness of the problem is affected not only by the number of DOFs but also
by the geometric meaning of the motion. By reformulating the problem to reduce
the number of DOFs, [38] and [33] increase the well-posedness of the task.
Whitening of warped events is proposed in [24] to mitigate event collapse,
while [23] designs effective regularizers based on the divergence or the area
deformation of the motion field, at the expense of increased computational
cost. Initialization close to the solution (in the basin of attraction of the
desired local optimum) can also play an important role in evading event
collapse [22].

Our work is most related to [23] because we focus on low-DOF motion estimation
tasks and seek a principled regularizer to gauge and penalize event collapse.
Our theoretical analysis provides a formula for homographic motions (8 DOFs),
which can be particularized for: 1 DOF (zoom-in/out motion), 2 DOFs (feature
flow), 3 DOFs (rotational camera motion), 4 DOFs (planar similarity motion)
and 6 DOFs (planar affine motion). While both [23] and our proposal are
interpretable and grounded on geometric principles of motion trajectories, the
most important theoretical difference is that our regularizer does not depend
on the event data. As a result, the regularizer drastically improves
computational complexity while achieving on par or better motion estimation
results on publicly-available datasets.

##  3 Methodology

This section first reviews how an event camera works (Section 3.1) as well as
the regularized CMax framework (Section 3.2). Then, we propose the new
regularizer and explain its geometrical meaning and implications (Section
3.3).

###  3.1 Event Camera

Instead of acquiring brightness images at fixed time intervals (e.g., frames),
event cameras record brightness differences asynchronously, called “events”
[1, 26]. An event $e_{k}\doteq(\mathbf{x}_{k},t_{k},p_{k})$ represents a
brightness change, and it is triggered as soon as the logarithmic brightness
at the pixel $\mathbf{x}_{k}\doteq(x_{k},y_{k})^{\top}$ exceeds a preset
threshold. Here, $t_{k}$ is the timestamp of the event with
$\mathrm{\SIUnitSymbolMicro s}$ resolution, and polarity $p_{k}\in\\{+1,-1\\}$
is the sign of the brightness change. Figure 2 shows the input events (red and
blue dots ($\mathbf{x}_{k},t_{k}$) in space-time, with color representing
polarity).

###  3.2 Regularized Contrast Maximization

The regularized CMax framework [23] aims at finding the motion parameters
$\boldsymbol{\theta}$ that optimize the objective function:

$\boldsymbol{\theta}^{\ast}=\arg\\!\min_{\boldsymbol{\theta}}\left(-G(\boldsymbol{\theta})+\lambda\mathcal{R}(\boldsymbol{\theta})\right).$
(1)

Here, $G$ is the data fidelity term and $\mathcal{R}$ is a regularizer with
weight $\lambda>0$. The overall steps of the framework are described in Figure
2, where black lines indicate the previous approaches, and blue lines indicate
our proposal.

####  3.2.1 Data fidelity term

The term $G(\boldsymbol{\theta})$ measures the alignment of the events with
respect to the candidate motion $\boldsymbol{\theta}$ (Figure 2, black solid
line). The original events $\mathcal{E}=\\{e_{k}\\}_{k=1}^{N_{e}}$ are
transformed according to the motion hypothesis into a set of warped events
$\mathcal{E}^{\prime}=\\{e^{\prime}_{k}\\}_{k=1}^{N_{e}}$:

$e_{k}\doteq(\mathbf{x}_{k},t_{k},p_{k})\quad\stackrel{{\scriptstyle\mathbf{W}}}{{\mapsto}}\quad
e^{\prime}_{k}\doteq(\mathbf{x}^{\prime}_{k},t_{\text{ref}},p_{k}).$ (2)

The warp function
$\mathbf{x}^{\prime}_{k}=\mathbf{W}(\mathbf{x}_{k},t_{k};\boldsymbol{\theta})$
transports every event along its motion trajectory until a reference time
$t_{\text{ref}}$ is reached. Point trajectories are parametrized by
$\boldsymbol{\theta}$, which consists of motion or scene unknowns (e.g., scene
depth, moving objects).

Powerful objective functions are designed based on the count of warped events
[25]. The representation of event count as an image is defined by the image of
warped events (IWE):

$I(\mathbf{x};\boldsymbol{\theta})\doteq\sum_{k=1}^{N_{e}}b_{k}\,\delta(\mathbf{x}-\mathbf{x}^{\prime}_{k}(\boldsymbol{\theta})).$
(3)

Each pixel of IWE counts how many events $e^{\prime}_{k}$ are warped into
pixel $\mathbf{x}$. The event polarity can be used by setting $b_{k}=p_{k}$,
and not used if $b_{k}=1$. The Dirac delta $\delta$ is approximated by a
Gaussian:
$\delta(\mathbf{x}-\boldsymbol{\mu})\approx\mathcal{N}(\mathbf{x};\boldsymbol{\mu},\epsilon^{2})$,
where $\epsilon=1$ pixel.

Finally, the objective function, such as the IWE variance is calculated:

$G(\boldsymbol{\theta})\equiv\operatorname{Var}\bigl{(}I(\mathbf{x};\boldsymbol{\theta})\bigr{)}\doteq\frac{1}{|\Omega|}\int_{\Omega}(I(\mathbf{x};\boldsymbol{\theta})-\mu_{I})^{2}d\mathbf{x},$
(4)

with mean
$\mu_{I}\doteq\frac{1}{|\Omega|}\int_{\Omega}I(\mathbf{x};\boldsymbol{\theta})d\mathbf{x}$
and image domain $\Omega$. The interpretation of (4) is as follows: the larger
the IWE variance (contrast of the IWE), the better the alignment of the warped
events $\mathcal{E}^{\prime}$. Contrast is related to sharpness and focus
[25].

####  3.2.2 Previous regularizers

The regularizer term $\mathcal{R}$ in (1) penalizes event collapse in certain
types of warps. When event collapse happens, the warped events are accumulated
into too few pixels or lines, resulting in an undesired global optimum $G$
(overfitting). Two regularizers are proposed in [23] by averaging some
collapse quantities attached to each event (Figure 2, dashed line).
Specifically, they use the divergence of the flow
$\mathcal{D}(\mathcal{E},\boldsymbol{\theta})$ and the area-based deformation
of the warp $\mathcal{A}(\mathcal{E},\boldsymbol{\theta})$, which are given
by:

$\begin{split}\mathcal{D}(\mathcal{E},\boldsymbol{\theta})&\doteq\\{\nabla\cdot\mathbf{f}_{k}\\}_{k=1}^{N_{e}},\\\
\mathcal{A}(\mathcal{E},\boldsymbol{\theta})&\doteq\bigl{\\{}|\det(\mathtt{J}(e_{k}))|\bigr{\\}}_{k=1}^{N_{e}},\end{split}$
(5)

where the flow
$\mathbf{f}\doteq\partial\mathbf{W}(\mathbf{x},t;\boldsymbol{\theta})/\partial
t$, and Jacobian of warp
$\mathtt{J}(\mathbf{x},t;\boldsymbol{\theta})\doteq\partial\mathbf{W}(\mathbf{x},t;\boldsymbol{\theta})/\partial\mathbf{x}$,
are the space-time derivatives of $\mathbf{W}$. Just like (3), $\mathcal{D}$
and $\mathcal{A}$ are used to create images of average divergence and area
deformation per pixel. Finally, [23] computes $\mathcal{R}$ as the trimmed
mean of such images.

###  3.3 Proposed Motion-based Regularizer

Although [23] successfully mitigates overfitting, it comes at a computational
cost. The complexity of these regularizers is $O(N_{e}+N_{p})$ because (5)
depends linearly on the number of events $N_{e}$ and the resulting average
images have $N_{p}$ pixels. This extra complexity makes the whole pipeline
more than twice slower than the original (unregularized) CMax framework, whose
complexity is also $O(N_{e}+N_{p})$ [22]. Not only the computational
complexity is a burden, but also the fact that (5) are measured relative to a
single reference time. For example,
$\mathcal{A}(\mathcal{E},\boldsymbol{\theta})$ increases as $t_{k}$ increases,
since it measures the area deformation _from $t_{k}$ to
$t_{\text{ref}}=t_{1}$_. This scaling problem is undesirable because ($i$)
events far from $t_{\text{ref}}$ contribute more to $\mathcal{R}$ than events
closer to $t_{\text{ref}}$, and ($ii$) this effect could be amplified
depending on the temporal distribution of the events.

Intuitively, motion fields are well-posed or not (i.e., collapse-enabled) by
design, regardless of the event data. Hence, an ideal regularizer should not
depend on the events, but solely on the warp parameters (Figure 2, blue line).
The main idea of the proposed regularizer is to aggregate differential
deformations rather than relative ones. Figure 3 shows the geometric
interpretation: $\mathcal{R}$ is obtained as the integral of the rate-of-
change of the area element deformation along the space-time point trajectories
$(\mathbf{x}(t),t)$ defined by the motion.

Figure 3: _Rate of change of area deformation._ The warp $\mathbf{W}$ defines
point trajectories $\boldsymbol{\gamma}(t)=(\mathbf{x}(t),t)$ in the space-
time image domain. We define the regularizer $\mathcal{R}$ based on
differential area deformation along $\boldsymbol{\gamma}(t)$. The rate of
change of area is given by the derivative of the Jacobian
$\mathtt{J}_{t,t+\Delta t}$.

####  3.3.1 Collapse-enabled warp with 1 DOF

To illustrate our approach, consider the simplest example: a 1-DOF motion that
approximates the translation of camera along its optical axis $Z$. This is a
simplified zoom-in/out motion without knowledge of scene depth, as used in
[34]. The warp $\mathbf{W}$ is given by

$\mathbf{x}^{\prime}_{k}=(1-t_{k}h_{z})\,\mathbf{x}_{k},$ (6)

where $\boldsymbol{\theta}\equiv h_{z}$, and the coordinate frame is at the
center of the image plane. For simplicity, $t\in[t_{1},t_{N_{e}}]$ is
normalized to $[0,1]$.

Assuming an area element attached to each point of the motion trajectory
$\boldsymbol{\gamma}(t)=(\mathbf{x}(t),t)$ (Figure 3), the change of area
(i.e., area deformation) from $t$ to $t+\Delta t$ is given by:

$|\mathtt{J}_{t,t+\Delta
t}|\doteq\left|\det\biggl{(}\frac{d\mathbf{x}(t+\Delta
t)}{d\mathbf{x}(t)}\biggr{)}\right|=\biggl{(}\frac{1-th_{z}}{1-(t+\Delta
t)h_{z}}\biggr{)}^{2}.$ (7)

The Taylor series expansion of (7) at $\Delta t=0$ is

$|\mathtt{J}_{t,t+\Delta t}|=1+\left.\frac{d|\mathtt{J}_{t,t+\Delta
t}|}{d\Delta t}\right|_{\Delta t=0}\Delta t+\cdots$ (8)

Since the first term is always 1 (i.e., is trivial), we focus on the second
term, which conveys the meaning of “speed” of area deformation. The derivative
of (7) at $\Delta t=0$ conveys the _rate of change_ or _differential
amplification factor_ of the area:

$\left.\frac{d|\mathtt{J}_{t,t+\Delta t}|}{d\Delta t}\right|_{\Delta
t=0}=\frac{2h_{z}}{1-th_{z}}.$ (9)

Finally, the total rate of change of the deformation along the observation
time window is

$\mathcal{R}\doteq\int_{0}^{1}\left.\frac{d|\mathtt{J}_{t,t+\Delta
t}|}{d\Delta t}\right|_{\Delta
t=0}dt\stackrel{{\scriptstyle\eqref{eq:rateofchange:oneDOF}}}{{=}}-2\log|1-h_{z}|.$
(10)

The regularizer (10) is plotted in Figure 4. It solely depends on
$\boldsymbol{\theta}\equiv h_{z}$ and has computational complexity $O(1)$. In
addition, it is developed from geometric principles, and it is interpretable:
$h_{z}=0$ (identity warp) gives $\mathcal{R}=0$; $h_{z}\in(0,1)$ (contraction;
collapsing warp) gives large $\mathcal{R}>0$; and $h_{z}<0$ (expansion warp)
gives $\mathcal{R}<0$. Moreover, notice that $\mathcal{R}$ behaves like a
barrier function, approaching infinity (i.e., large penalty) for values close
to $h_{z}=1$ (maximum allowed contraction before events flip side with respect
to the image center).

Figure 4: Regularizer $\mathcal{R}$ for the 1-DOF warp, (10).

####  3.3.2 Well-posed warp with 2 DOFs

The 2-DOF translational motion (feature flow) in image space is a well-posed
warp, since collapse never happens because the motion lines are parallel. The
warp is given by
$\mathbf{W}(\mathbf{x},t;\boldsymbol{\theta})=\mathbf{x}+(t-t_{\text{ref}})\boldsymbol{\theta}$
with $\boldsymbol{\theta}\equiv(v_{x},v_{y})^{\top}$, assuming constant
velocity $\boldsymbol{\theta}$ for all pixels. Translations do not change the
area element, i.e., $\mathtt{J}_{t,t+\Delta t}=1$, hence
$\mathcal{R}\equiv\int_{0}^{1}\frac{d|\mathtt{J}|}{d\Delta t}=0$. Since the
resulting regularizer vanishes, it does not affect the landscape of the
composite objective function (1), as expected.

####  3.3.3 Well-posed warp with 3 DOFs

Angular velocity estimation of a purely rotating event camera is a common
research problem [28, 33, 24, 27]. This 3-DOF motion is another well-posed
warp, although it involves small amounts of deformation [23]. Assuming a small
time interval, the warp is parametrized by the angular velocity
$\boldsymbol{\theta}\equiv\boldsymbol{\omega}=(\omega_{x},\omega_{y},\omega_{z})^{\top}$.
In calibrated homogeneous coordinates $\mathbf{x}^{h}$, the point trajectories
are described by [22]:

$\mathbf{x}^{h}(t)\sim\mathtt{R}(\boldsymbol{\omega}t)\,\mathbf{x}^{h}(0),$
(11)

where $t\in[0,1]$ is normalized and
$\mathtt{R}(\boldsymbol{\phi})\doteq\exp(\boldsymbol{\phi}^{\wedge})$ is
parametrized using exponential coordinates [39, 40].

The incremental rotation between $t$ and $t+\Delta t$ yields

$\mathbf{x}^{h}(t+\Delta t)\sim\mathtt{R}(\boldsymbol{\omega}\Delta
t)\,\mathbf{x}^{h}(t).$ (12)

TABLE I: Results on the MVSEC dataset [41]. The best values per column per
group are in bold, and the second best are underlined. An asterisk in FWL
indicates that event collapse occurred.

Variance Gradient Magnitude AEE $\downarrow$ 3PE $\downarrow$ 10PE
$\downarrow$ 20PE $\downarrow$ FWL $\uparrow$ AEE $\downarrow$ 3PE
$\downarrow$ 10PE $\downarrow$ 20PE $\downarrow$ FWL $\uparrow$ Ground truth
flow – – – – $1.047\,993\,099$ – – – – $1.047\,993\,099$ Identity warp
$4.845\,102\,657$ $60.585\,450\,33$ $10.384\,180\,53$ $0.314\,555\,551$ $1.0$
$4.845\,102\,657$ $60.585\,450\,33$ $10.384\,180\,53$ $0.314\,555\,551$ $1.0$
1 DOF No regularizer $89.339\,76$ $97.299\,11$ $95.417\,347\,07$
$92.390\,566\,33$ ∗1.90 $85.774\,515$ $93.964\,33$ $86.235\,230\,74$
$83.445\,622\,01$ ∗1.87 Whitening [24] $89.5848$ $97.181\,28$
$96.772\,802\,41$ $93.760\,300\,44$ ∗1.90 $81.101\,769\,19$ $90.860\,110\,26$
$89.043\,858\,9$ $86.200\,254\,93$ ∗1.85 Deformation [23] $4.465\,84$
$52.600\,07$ $5.162\,230\,828$ $0.127\,030\,408$ $1.0766$ $3.970\,13$
$48.7886$ $3.209\,162\,316$ $0.066\,383\,923\,14$ $1.0873$ Div. + Def. [23]
$3.299\,381$ $33.087\,535\,45$ $2.613\,900\,583$ $0.484\,262\,705\,5$
$1.196\,874\,476$ $2.849\,103\,044$ $32.337\,056\,58$ $2.435\,868\,209$
$0.032\,464\,283\,93$ $1.174\,927\,461$ Ours $3.167\,840\,37$
$36.654\,285\,35$ $4.010\,612\,447$ $0.095\,951\,733\,36$ $1.155\,540\,142$
$3.023\,115\,819$ $34.469\,039\,54$ $3.395\,991\,12$ $0.070\,931\,795\,77$
$1.165\,741\,409$ 4 DOF No regularizer $90.215\,397\,7$ $90.215\,397\,7$
$96.940\,675\,2$ $93.864\,536\,51$ ∗2.05 $91.263\,285\,56$ $99.485\,319\,96$
$95.061\,646\,99$ $91.463\,053\,09$ ∗2.01 Whitening [24] $90.818\,409\,15$
$99.109\,810\,7$ $98.037\,577\,81$ $95.041\,879\,6$ ∗2.04 $88.384\,647\,66$
$98.869\,765\,57$ $92.411\,993\,18$ $88.663\,739\,37$ ∗2.00 Deformation [23]
$8.132\,825\,707$ $87.459\,245\,6$ $18.531\,339\,53$ $1.092\,239\,517$
$1.027\,891\,481$ $5.253\,162\,047$ $64.787\,401\,94$ $13.175\,004\,67$
$0.373\,906\,813\,8$ $1.151\,436\,025$ Div. + Def. [23] $5.144\,825\,74$
$65.607\,465\,89$ $10.754\,696\,45$ $0.381\,769\,796\,2$ $1.157\,302\,645$
$5.409\,725\,124$ $66.014\,200\,62$ $13.188\,804\,75$ $0.541\,780\,113\,6$
$1.143\,399\,47$ Ours $4.359\,446\,773$ $58.633\,103\,97$ $6.556\,431\,157$
$0.156\,884\,346\,9$ $1.148\,671\,166$ $4.300\,043\,269$ $54.268\,809\,05$
$5.612\,462\,037$ $0.309\,605\,148\,8$ $1.173\,120\,008$

TABLE II: Results on the DSEC dataset [42]. Same notation as Table I.

Variance Gradient Magnitude AEE $\downarrow$ 3PE $\downarrow$ 10PE
$\downarrow$ 20PE $\downarrow$ FWL $\uparrow$ AEE $\downarrow$ 3PE
$\downarrow$ 10PE $\downarrow$ 20PE $\downarrow$ FWL $\uparrow$ Ground truth
flow – – – – $1.090\,988\,475$ – – – – $1.090\,988\,475$ Identity warp
$5.843\,080\,683$ $60.453\,759\,86$ $16.645\,577\,3$ $3.395\,154\,843$ $1.0$
$5.843\,080\,683$ $60.453\,759\,86$ $16.645\,577\,3$ $3.395\,154\,843$ $1.0$ 1
DOF No regularizer $156.125\,839\,4$ $99.883\,541\,01$ $99.333\,097\,76$
$98.181\,406\,73$ ∗2.58 $156.077\,905\,5$ $99.927\,589\,34$ $99.401\,588\,47$
$98.108\,422\,33$ ∗2.58 Whitening [24] $156.178\,465\,3$ $99.952\,860\,01$
$99.514\,527\,98$ $98.261\,199\,02$ ∗2.58 $156.817\,522\,3$ $99.879\,812\,64$
$99.383\,526\,44$ $98.329\,376\,88$ ∗2.58 Deformation [23] $9.007\,609\,466$
$68.956\,085\,51$ $18.861\,185\,38$ $4.773\,455\,209$ $1.402\,945\,911$
$5.793\,269\,017$ $64.022\,456\,64$ $16.108\,408\,44$ $2.751\,044\,423$
$1.361\,735\,211$ Div. + Def. [23] $6.061\,261\,761$ $68.479\,370\,73$
$17.084\,346\,78$ $2.272\,174\,046$ $1.356\,512\,051$ $5.526\,156\,883$
$64.093\,066\,51$ $15.064\,542\,71$ $1.368\,051\,902$ $1.348\,987\,971$ Ours
$5.805\,870\,209$ $57.188\,200\,89$ $14.734\,544\,5$ $3.052\,300\,535$
$1.340\,454\,148$ $5.306\,213\,279$ $54.851\,182\,15$ $14.172\,296\,52$
$3.104\,782\,764$ $1.200\,084\,868$ 4 DOF No regularizer $157.542\,172$
$99.968\,215\,14$ $99.642\,887\,96$ $98.666\,985\,69$ ∗2.64 $157.335\,601\,7$
$99.942\,549\,34$ $99.525\,213\,17$ $98.444\,496\,33$ ∗2.62 Whitening [24]
$157.732\,983\,9$ $99.969\,084\,4$ $99.661\,319\,66$ $98.713\,352\,43$ ∗2.60
$156.118\,086\,8$ $99.912\,184\,62$ $99.258\,211\,52$ $97.929\,470\,05$ ∗2.61
Deformation [23] $15.118\,467\,67$ $94.955\,666\,48$ $62.585\,701\,84$
$22.616\,035\,5$ $1.247\,289\,93$ $10.006\,316$ $90.146\,151\,44$
$39.451\,494\,29$ $8.670\,040\,138$ $1.249\,295\,937$ Div. + Def. [23]
$10.059\,025\,13$ $90.646\,871\,98$ $40.607\,431\,78$ $8.583\,359\,441$
$1.263\,127\,072$ $10.393\,353\,83$ $91.018\,175\,47$ $41.812\,122\,68$
$9.396\,431\,802$ $1.226\,296\,549$ Ours $11.514\,379\,75$ $91.501\,522\,07$
$42.288\,367\,3$ $11.052\,171\,35$ $1.296\,983\,665$ $9.554\,259\,428$
$88.940\,764\,93$ $35.958\,205\,72$ $7.742\,679\,675$ $1.309\,339\,262$

Hence, the area element at $\mathbf{x}(t)$ deforms according to:

$\det\biggl{(}\frac{d\mathbf{x}(t+\Delta
t)}{d\mathbf{x}(t)}\biggr{)}=\bigl{(}\mathbf{r}_{3}^{\top}(\boldsymbol{\omega}\Delta
t)\mathbf{x}^{h}(t)\bigr{)}^{-3},$ (13)

where $\mathbf{r}_{3}^{\top}$ is the third row of
$\mathtt{R}(\boldsymbol{\omega}\Delta t)$ (see Section A.1). The derivative of
(13) at $\Delta t=0$ is given by:

$\left.\frac{d|\mathtt{J}_{t,t+\Delta t}|}{d\Delta t}\right|_{\Delta
t=0}=3\mathbf{x}^{h\top}(t)\boldsymbol{\omega}^{\wedge}\mathbf{e}_{3}.$ (14)

Finally, the integral of (14) over the point trajectory (parametrized by the
initial point $\mathbf{x}(0)$) is given by:

$\begin{split}\mathcal{R}_{\mathbf{x}(0)}&=\int_{0}^{1}\left.\frac{d|\mathtt{J}_{t,t+\Delta
t}|}{d\Delta t}\right|_{\Delta t=0}dt\\\
&=3\omega_{x}\int_{0}^{1}y(t)dt-3\omega_{y}\int_{0}^{1}x(t)dt.\end{split}$
(15)

The integrals in (15) have units of absement. To obtain the regularizer
$\mathcal{R}$, we threshold $\mathcal{R}_{\mathbf{x}(0)}$ at $-0.2$ and
compute its mean, which allows small amounts of natural deformation caused by
rotation. Similar to (10), (15) does not depend on the events. However, in
contrast to (10), (15) is spatially varying, providing an aggregated
deformation map: it is smaller in the center of the image and larger (in
absolute value) in the periphery. The computational complexity of
$\mathcal{R}$ is $O(N_{p})$, which can be further reduced if only a subset of
the pixels is used.

Although 3-DOF rotations involve small deformations, their values (14) are
considerably smaller than those of collapse-enabled warps like (6), and
$\mathcal{R}$ does not affect the accuracy of the angular velocity estimation
(as Section 4.4 will show). Also, pure rotations around the $Z$ axis
$\boldsymbol{\omega}=(0,0,\omega_{z})^{\top}$ do not change the area, as
expected, resulting in $\mathcal{R}=0$.

####  3.3.4 Collapse-enabled warp with 4 DOFs

The 1-DOF warp (Section 3.3.1) is a particular case of the 4-DOF warp in [34,
24], which approximates a freely-moving camera (6 DOFs) by means of a
similarity transformation on the image plane. The scaling parameter $h_{z}$ of
the similarity transformation controls the amount of zoom in/out, i.e., the
amount of contraction/expansion of the warp. Hence, we use (10) to penalize
the amount of contraction. A mathematical justification is given in Appendix
A. Since the purpose of our regularizer is to discourage the collapse while
allowing a small amount of natural collapse, we use
$\mathcal{R}=\max\left(\alpha,-2\log|1-h_{z}|\right)-\alpha$, with margin
$\alpha=1$.

##  4 Experiments

|  _Original_ |  | _Without regularization_ |  |  _With regularization_
---|---|---|---|---|---
MVSEC |  |  |  |  |  |  |
DSEC |  |  |  |  |  |  |
boxes_rot |  |  |  |  |  |  |
dynamic_rot |  |  |  |  |  |  |
|  (a) Input events |  |  (b) Best IWE |  (c) Deformation [23] |   (d) Ours |  |  (e) Best IWE
Figure 5: _Qualitative results._ (a) Original events. (b)-(d) Results without
regularization: 1-DOF motion results (MVSEC [43] and DSEC [42]) are trapped in
global optima of event collapse, as shown in the IWEs (b). The regularizers in
such collapse cases (c)-(d) are very large compared with the well-posed warp
cases (boxes_rot and dynamic_rot rows). (e) Results with the proposed
regularizer: it mitigates collapse for MVSEC and DSEC scenes while it does not
harm the ECD scenes. Best viewed in the electronic version.

We assess the performance of our regularizer by first showing its
effectiveness on collapse-enabled warps, which naturally appear in driving
sequences. Second, a runtime analysis proves that our proposal is faster than
prior work. Finally, we also demonstrate the effect on rotational sequences,
in order to show that the proposed regularizer does not harm well-posed warps.
There is no need to test feature flow since we have proved analytically that
the regularizer vanishes for such a warp.

###  4.1 Setup: Datasets and Evaluation Metrics

####  4.1.1 Datasets

The _MVSEC_ dataset [43] is a standard dataset to evaluate various tasks, such
as optical flow estimation [14, 44, 15, 16]. The dataset consists of events,
grayscale frames and IMU data from an event camera (mDAVIS346, $346\times 260$
pixels [45]), camera poses, LiDAR data, and ground truth optical flow provided
by [41]. We use the outdoor sequences, where the event camera is mounted on a
car. Following previous work [23], we select several excerpts from the
_outdoor_day1_ sequence that have a dominant forward motion, which is
reasonably well approximated by collapse-enabled warps such as 1 DOF and 4 DOF
cases. In total, we evaluate on 3.2 million events spanning 10 s.

The _DSEC_ dataset [42] is another recent dataset of driving sequences. It
includes more complex scenes (e.g., moving objects, higher dynamic range) with
a higher resolution event camera (Prophesee Gen3, $640\times 480$ pixels).
Ground truth optical flow is computed as the motion field, using the scene
depth from a LiDAR [44]. In total, we evaluate on 380 million events spanning
40 s from the _zurich_city_11_ sequence.

The _ECD_ dataset [46] is widely used to assess camera ego-motion [28, 9, 11,
27, 8, 47, 10, 23]. Each sequence provides events, frames, calibration
information, and IMU data from a DAVIS240C camera ($240\times 180$ pixels
[48]), as well as ground truth camera poses from a motion capture system (at
200Hz). We use _boxes_rotation_ and _dynamic_rotation_ sequences for 3-DOF
rotational motion estimation, to consistently compare with previous work. In
total, we evaluate on 43 million events (10 s) of the box sequence, and on 15
million events (11 s) of the dynamic sequence.

####  4.1.2 Metrics

Optical flow accuracy (for MVSEC and DSEC experiments) is given by the Average
Endpoint Error (AEE) and the percentage of pixels with AEE greater than $N$
pixels (“$N$PE”), for $N=\\{3,10,20\\}$. They are calculated only in pixels
with valid ground truth. We also adopt the FWL metric [49], which is defined
as the relative variance of the IWE with respect to that of the identity warp.
The FWL measures the IWE sharpness: FWL $<1$ means that the estimation is
worse than the zero-flow baseline, while FWL $>1$ implies that the resulting
IWE is sharper than the baseline.

Rotational motion accuracy is assessed as the RMS error of angular velocity
estimation, following previous works [25, 24, 27]. Angular velocity is assumed
to be constant during the time window of events, and compared with the IMU’s
gyroscope value (ground truth) at the midpoint: $(t_{1}+t_{N_{e}})/2$. We also
use the FWL metric to measure the IWE sharpness.

The estimation time window spans: $dt=4$ grayscale frames (at $\approx$ 45Hz)
in MVSEC (standard for MVSEC benchmark), 500k events for DSEC, and 30k events
for ECD dataset, respectively. For runtime comparison we use a fixed number of
events (30k for MVSEC and 500k for DSEC), because the runtime depends on the
number of events (e.g., $O(N_{e}+N_{p})$). We set $\lambda$ in (1) as follows:
if the data term is the IWE variance, $\lambda=1.0$ for MVSEC and ECD
experiments, and $\lambda=5.0$ for DSEC experiments; if the data term is the
squared magnitude of the IWE gradient, $\lambda=0.2$ for MVSEC and
$\lambda=1.0$ for DSEC experiments. The optimization algorithm is the Tree-
Structured Parzen Estimator (TPE) sampler [50].

###  4.2 Results on Collapse-Enabled Warps

Tables I and II report the results of collapse-enabled warp experiments (1 and
4 DOFs) on MVSEC and DSEC, respectively. They report the flow AEE, $N$PE, and
FWL for the two data terms: image variance and the squared magnitude of the
IWE gradient (“Gradient Magnitude”). Throughout the experiments, both metrics
are considerably high in the results of the original CMax (“No regularizer”)
and whitening [24] methods. This indicates that the warp overfits to events
(event collapse). On the other hand, our regularizer produces better AEE
values and moderately higher FWL than $1$. These results clearly show that our
regularizer successfully discourages event overfitting while producing sharper
IWEs than the identity warp. Our results are competitive with those by [23],
and we do not find significant accuracy differences between them.

Qualitative results are shown in Figure 5 (MVSEC and DSEC rows). Our
regularizer (last column) provides the best IWEs, which reveal the sharp edges
of the scene, while the IWEs without regularizer produce event collapse
(second column). Although $\mathcal{R}$ in (10) is a scalar, we visualize it
as an image to compare it with the area deformation map from [23]. Notice that
the area deformation map [23] shows collapse _only_ at pixels with warped
events, while our regularizer provides dense maps (even in pixels with no
events, corresponding to homogeneous brightness regions) because it is purely
geometric, based on the motion parameters.

###  4.3 Runtime Comparison

Table III reports the runtime comparison of the methods, notably with respect
to the original CMax (“No regularizer”). We use Python (3.9.12) on a CPU (Mac
M1 2020, 8 Cores), and average the runtime over 400 trials. The whitening
technique [24] is slower than the original CMax (“No regularizer”). The
runtime difference is due to an extra SVD step on the events, which is more
noticeable ($2\times$ slower) in the DSEC dataset than in MVSEC because it
uses more events. The “Deformation” regularizer in [23] is also two to three
times slower than the original CMax. When both regularizers in [23] are
combined (“Div. + Def.”), the runtime becomes even larger. Finally, our
regularized approach has almost the same runtime as the original CMax, since
its complexity is $O(1)$, thus it is two to four times faster than competing
methods.

Figure 6 visualizes the accuracy and runtime of the methods (on DSEC data).
Runtime is reported relative to the “No regularizer” case. It clearly shows
that the proposed regularizer is the only effective approach against event
collapse that does not compromise the speed of the CMax framework.

TABLE III: Comparison of runtime (in milliseconds), averaged over 400 trials.
MVSEC: 30k events. DSEC: 500k events.

MVSEC DSEC Var. Grad. Var. Grad. No regularizer $7.27$ $7.91$ $111.285$
$112.8775$ Whitening [24] $8.225$ $8.36$ $205.4625$ $206.89$ Deformation [23]
$20.22$ $21.1375$ $304.44$ $307.6225$ Div. + Def. [23] $32.4075$ $31.6225$
$505.02$ $506.1425$ Ours $7.36$ $8.025$ $111.4$ $113.51$

Figure 6: _Runtime comparison_ for the DSEC experiment. Runtime is relative to
that of the original CMax (“No regularizer”). Our method has desirable
properties: small AEE and runtime.

###  4.4 Results on Well-Posed Warps

To confirm that the proposed regularizer does not harm well-posed warps (e.g.,
3-DOF rotational motion), we report results on the ECD dataset in Table IV. We
use the variance as data fidelity and the Adam optimizer. In both rotational
sequences, the results of “No regularizer” and ours produce very similar RMS
and FWL values. This is because $\mathcal{R}$ values are small and do not
affect the landscape of the objective function. As qualitatively shown in
Figure 5, for the 3 DOF experiments the absolute values of
$\mathcal{R}_{\mathbf{x}(0)}$ are approximately less than $0.1$, while for the
collapse-enabled warps they are larger than $10$ (i.e., two orders of
magnitude difference).

TABLE IV: Results on ECD dataset [46].

boxes_rot dynamic_rot RMS $\downarrow$ FWL $\uparrow$ RMS $\downarrow$ FWL
$\uparrow$ Ground truth pose – $1.559\,016\,91$ – $1.414\,303\,624$ No
regularizer $8.857\,855\,6$ $1.561\,590\,243$ $4.823\,395\,392$
$1.420\,484\,652$ Deformation [23] $8.664\,275\,512$ $1.560\,753\,596$
$4.822\,492\,508$ $1.420\,484\,1$ Div. + Def. [23] $6.885\,339\,903$
$1.561\,584\,869$ $4.821\,954\,2$ $1.420\,483\,918$ Ours $6.877\,290\,364$
$1.561\,589\,855$ $4.821\,937\,829$ $1.420\,482\,71$

###  4.5 Application: Time-to-contact

The parametrization of collapse-enabled warps has useful implications toward
future application on intelligent vehicles, such as advanced driver-assistance
system (ADAS). Let us introduce another interpretation of the parameter
$h_{z}$. For a freely moving camera with linear and angular velocities
$\mathbf{V}$ and $\boldsymbol{\omega}$, respectively, the apparent velocity
$\mathbf{v}(\mathbf{x})$ on the image plane of a 3D point
$\mathbf{X}=(x,y,Z(\mathbf{x}))^{\top}$ (at depth $Z(\mathbf{x})$ with respect
to the camera) can be computed using the $2\times 6$ feature sensitivity
matrix [51]:

$\mathbf{v}(\mathbf{x})=\begin{pmatrix}\frac{-1}{Z(\mathbf{x})}&0&\frac{x}{Z(\mathbf{x})}&xy&-(1+x^{2})&y\\\\[4.30554pt]
0&\frac{-1}{Z(\mathbf{x})}&\frac{y}{Z(\mathbf{x})}&1+y^{2}&-xy&x\\\
\end{pmatrix}\\!\\!\begin{pmatrix}\mathbf{V}\\\\[4.30554pt]
\boldsymbol{\omega}\\\ \end{pmatrix},$ (16)

which can be used to warp events:

$\mathbf{x}_{k}^{\prime}=\mathbf{x}_{k}-\mathbf{v}(\mathbf{x})t_{k}.$ (17)

Assuming a vehicle with body-frame velocity $v_{z}$, i.e.,
$\mathbf{V}\equiv(0,0,v_{z})^{\top}$,
$\boldsymbol{\omega}\equiv(0,0,0)^{\top}$, the motion field (16) becomes
$\mathbf{v}(\mathbf{x})=(v_{z}/Z(\mathbf{x}))\,\mathbf{x}$, and substituting
in (17) gives $\mathbf{x}_{k}^{\prime}=(1-v_{z}/Z(\mathbf{x}))t_{k}.$
Comparing this expression to (6), and assuming $Z(\mathbf{x})$ is spatially
invariant, we identify

$h_{z}=\frac{v_{z}}{Z},$ (18)

i.e., the parameter $h_{z}$ is inverse of the time-to-contact or time-to-
collision (TTC) [52].

Figure 7: _Time to Contact_ application example. The parametrization with
$h_{z}$ in the 1-DOF warp can be used to approximate the TTC for the dominant
depth of the scene represented by the events (e.g., the trees).

Figure 7 shows two examples of TTC from the MVSEC dataset. It is remarkable
that this 1-DOF warp model can be related to the popular concept in ADAS, and
our regularizer plays an important role toward real-time computation of TTC
given its runtime. Also note that (18) establishes a relation between TTC,
vehicle speed and scene depth, and that the TTC can be used to estimate the
scene depth given the vehicle speed, or vice versa, the vehicle speed given
the scene depth. We hope this connection helps future implementation of event-
camera application in collision avoidance systems.

##  5 Limitations

As in many regularized problems, the regularizer weight $\lambda$ is
empirically set. It depends on the paired objective function and on the data,
i.e., on the scene. The proposed method also has a heuristically selected
threshold (margin) that allows us to account for small natural deformations of
the motion field. It would be desirable to develop automatic strategies to
balance both data fidelity and regularizing terms (including the threshold)
for best optimization convergence and results.

While we have obtained formulas to measure the rate of change of area
deformation along point trajectories for low-DOF motions (up to 8-DOF
homographies in Appendix A), the regularizer requires aggregation in the form
of integrals (e.g., (15)), which are evaluated via numerical integration.
Extending the ideas in this work to higher DOFs (e.g., optical flow) and
developing efficient numerical approaches is an important research direction.

Our current implementation of the regularized CMax framework is not real time.
This is especially relevant as the spatial resolution of event cameras
increases (VGA size [42] and 1Mpixel event cameras [2]), which also increases
the number of events to be processed. Therefore, in the future it will be
important to speed up the method (both the data fidelity term and the
regularizer) to enable interactive applications.

Finally, while in the time-to-contact application (Section 4.5) we established
the connection between scene depth and vehicle speed, the example warp is
1-DOF (assumes a single depth for the whole scene), which may be an over-
simplification. The warp does not consider where events happen from a user
perspective (e.g., on the road or on the side of the road). A more elaborate
model would consider the location of the events and decide whether there will
be an actual contact along the predicted vehicle trajectory.

##  6 Conclusion

We proposed a novel regularizer to mitigate event collapse in the CMax
framework, based on aggregating differential deformations of the motion field.
The experimental results show its efficacy, achieving on-par state-of-the-art
accuracy in low-DOF motion estimation problems. Furthermore, the proposed
regularizer is the only effective approach to date against event collapse that
does not compromise the runtime of the CMax framework. Since the low-DOF
motion estimation forms a foundation of more complex motion models, it would
be important to analyze more complex warps, such as dense optical flow, to
alleviate event collapse effectively and efficiently. We hope this work
encourages future research towards this paramount application of the CMax
framework that leverages the advantages of event cameras.

## Appendix A

In homogeneous coordinates, a homographic warp $\mathbf{W}$ is given by [22]

$\mathbf{x}^{h\prime}_{k}\sim\mathtt{H}^{-1}(t_{k};\boldsymbol{\theta})\,\mathbf{x}^{h}_{k},$
(19)

and the point trajectories are represented by

$\mathbf{x}^{h}(t)\sim\mathtt{H}(t;\boldsymbol{\theta})\,\mathbf{x}^{h}(0).$
(20)

Hence, the differential transformation from $t$ to $t+\Delta t$ is given also
by a homography $\mathtt{H}_{t,t+\Delta t}$:

$\mathbf{x}^{h}(t+\Delta t)\sim\underbrace{\mathtt{H}(t+\Delta
t;\boldsymbol{\theta})\,\mathtt{H}^{-1}(t;\boldsymbol{\theta})}_{\mathtt{H}_{t,t+\Delta
t}}\mathbf{x}^{h}(t).$ (21)

Using Result 1 in Appendix A of [12], the determinant of the Jacobian
$\mathtt{J}$ of the transformation (from $t$ to $t+\Delta t$) in Euclidean
coordinates is

$\det\bigl{(}\mathtt{J}_{t,t+\Delta
t}\bigr{)}=\frac{\det(\mathtt{H}_{t,t+\Delta
t})}{(\mathbf{e}_{3}^{\top}\mathtt{H}_{t,t+\Delta
t}\,\mathbf{x}^{h}(t))^{3}},$ (22)

where $\mathbf{e}_{3}=(0,0,1)^{\top}$,
$\mathbf{x}^{h}(t)=(x(t),y(t),1)^{\top}$.

The 8-DOF motion admits several particular cases.

###  A.1 Rotational Camera Motion

Here,
$\mathtt{H}(t;\boldsymbol{\theta})\equiv\mathtt{R}(t\boldsymbol{\omega})$ in
(20) is a $3\times 3$ rotation matrix. Rotation matrices have unit determinant
and simplify: $\mathtt{H}_{t,t+\Delta t}=\mathtt{R}((t+\Delta
t)\boldsymbol{\omega})\,\mathtt{R}^{-1}(t\boldsymbol{\omega})=\exp\bigl{(}((t+\Delta
t)\boldsymbol{\omega})^{\wedge}\bigr{)}\exp\bigl{(}(-t\boldsymbol{\omega})^{\wedge}\bigr{)}=\exp((\Delta
t\,\boldsymbol{\omega})^{\wedge})=\mathtt{R}(\boldsymbol{\omega}\Delta t)$.
This holds because the rotation axis $\boldsymbol{\omega}$ is unique. Hence
(22) becomes:

$\det\bigl{(}\mathtt{J}_{t,t+\Delta
t}\bigr{)}=\frac{1}{(\mathbf{e}_{3}^{\top}\mathtt{R}(\boldsymbol{\omega}\Delta
t)\,\mathbf{x}^{h})^{3}},$ (23)

which is (13). Computing the derivative yields (14):

$\displaystyle\left.\frac{d|\mathtt{J}_{t,t+\Delta t}|}{d\Delta
t}\right|_{\Delta t=0}$
$\displaystyle=\frac{-3\mathbf{x}^{h\top}(t)}{\bigl{(}\mathbf{r}_{3}^{\top}(\boldsymbol{\omega}\Delta
t)\mathbf{x}^{h}(t)\bigr{)}^{4}}\,\frac{d}{d\Delta
t}\mathbf{r}_{3}(\boldsymbol{\omega}\Delta t)\bigg{|}_{\Delta t=0}$
$\displaystyle=3\mathbf{x}^{h\top}(t)\boldsymbol{\omega}^{\wedge}\mathbf{e}_{3},$
(24)

because

$\begin{split}\frac{d}{d\Delta t}\mathbf{r}_{3}(\boldsymbol{\omega}\Delta
t)&=\frac{d}{d\Delta t}\mathtt{R}^{\top}(\boldsymbol{\omega}\Delta
t)\mathbf{e}_{3}\\\ &\approx\frac{d}{d\Delta
t}\bigl{(}\mathtt{Id}-(\boldsymbol{\omega}\Delta
t)^{\wedge}\bigr{)}\mathbf{e}_{3}\\\
&=-\boldsymbol{\omega}^{\wedge}\mathbf{e}_{3}\\\
&=(-\omega_{y},\omega_{x},0)^{\top}.\end{split}$ (25)

###  A.2 Similarity Transformation on the Image Plane,
$\displaystyle\operatorname{Sim}(2)$

A planar similarity transformation has 4 DOFs and can be parametrized using
linear image velocity, angular velocity around $Z$, and scaling speed on the
image plane, $\boldsymbol{\theta}=(\mathbf{v},\omega_{z},s)^{\top}$ [24]. The
point trajectories are given by (20) with

$\mathtt{H}^{S}(t;\boldsymbol{\theta})\doteq\begin{pmatrix}\beta(t;s)\mathtt{R}(t\omega_{z})&t\mathbf{v}\\\
\mathbf{0}^{\top}&1\end{pmatrix}.$ (26)

Using (21) it gives $\mathtt{H}_{t,t+\Delta t}=\mathtt{H}^{S}(t+\Delta
t;\boldsymbol{\theta})\,(\mathtt{H}^{S})^{-1}(t;\boldsymbol{\theta})$.
Similarities form a matrix Lie group, hence the inverse and the product of two
similarities is also a similarity. Since $\mathtt{H}_{t,t+\Delta t}$ is a
similarity, its third row is $\mathbf{e}_{3}^{\top}\mathtt{H}_{t,t+\Delta
t}=(0,0,1)$, which makes the denominator in (22) equal to one. Substituting in
(22) produces

$\begin{split}|\mathtt{J}_{t,t+\Delta
t}|&=\left|\det\bigl{(}\mathtt{H}^{S}(t+\Delta
t;\boldsymbol{\theta})\,(\mathtt{H}^{S})^{-1}(t;\boldsymbol{\theta})\bigr{)}\right|\\\\[2.15277pt]
&=\left|\frac{\det\bigl{(}\mathtt{H}^{S}(t+\Delta
t;\boldsymbol{\theta})\bigr{)}}{\det\bigl{(}\mathtt{H}^{S}(t;\boldsymbol{\theta})\bigr{)}}\right|\\\
&=\left(\frac{\beta(t+\Delta t;s)}{\beta(t;s)}\right)^{2}.\end{split}$ (27)

Two intuitive remarks about (27): ($i$) it only depends on the scaling DOF
(i.e., one out of the four DOFs) since $\omega_{z}$ and $\mathbf{v}$ do not
appear; and ($ii$) it can be used to derive the 1-DOF formula (7): the 1-DOF
scaling transformation is modeled by $\mathtt{H}^{S}$ with
$\omega_{z}=0,\mathbf{v}=\mathbf{0}$ and $\beta(t;h_{z})=(1-th_{z})^{-1}$.
Substituting this choice of $\beta$ in (27) makes it coincide with (7).

The 4-DOF transformation in [34] has a similar geometric meaning but a
different parametrization. Hence, we use the above result and penalize
collapse by means of the corresponding scaling parameter in [34].

###  A.3 Affine Transformation on the Image Plane,
$\displaystyle\operatorname{Aff}(2)$

A planar affine transformation has 6 DOFs in $\boldsymbol{\theta}$. Letting

$\mathtt{H}^{A}(t;\boldsymbol{\theta})\doteq\begin{pmatrix}\mathtt{A}(t;\boldsymbol{\theta})&t\mathbf{b}\\\
\mathbf{0}^{\top}&1\end{pmatrix},$ (28)

and using (21) gives $\mathtt{H}_{t,t+\Delta t}=\mathtt{H}^{A}(t+\Delta
t;\boldsymbol{\theta})\,(\mathtt{H}^{A})^{-1}(t;\boldsymbol{\theta})$.
Affinities also form a matrix Lie group, hence $\mathtt{H}_{t,t+\Delta t}$ is
an affinity. Moreover, its third row is also
$\mathbf{e}_{3}^{\top}\mathtt{H}_{t,t+\Delta t}=(0,0,1)$, and following
similar steps as those in (27) yields

$\begin{split}|\mathtt{J}_{t,t+\Delta
t}|&=\left|\det\bigl{(}\mathtt{H}^{A}(t+\Delta
t;\boldsymbol{\theta})\,(\mathtt{H}^{A})^{-1}(t;\boldsymbol{\theta})\bigr{)}\right|\\\\[2.15277pt]
&=\left|\frac{\det\bigl{(}\mathtt{A}(t+\Delta
t;\boldsymbol{\theta})\bigr{)}}{\det\bigl{(}\mathtt{A}(t;\boldsymbol{\theta})\bigr{)}}\right|.\end{split}$
(29)

Notice that the $2\times 2$ matrix $\mathtt{A}$ includes not only a scaling
parameter but also a shear, which affects the area deformation.

## Acknowledgments

This research was funded by the German Academic Exchange Service (DAAD),
Research Grant-Bi-nationally Supervised Doctoral Degrees/Cotutelle, 2021/22
(57552338) and the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – EXC 2002/1 “Science of
Intelligence” – project number 390523135.

## References

  * [1] P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128$\times$128 120 dB 15 $\mu$s latency asynchronous temporal contrast vision sensor,” _IEEE J. Solid-State Circuits_ , vol. 43, no. 2, pp. 566–576, 2008.
  * [2] T. Finateu, A. Niwa, D. Matolin, K. Tsuchimoto, A. Mascheroni, E. Reynaud, P. Mostafalu, F. Brady, L. Chotard, F. LeGoff, H. Takahashi, H. Wakabayashi, Y. Oike, and C. Posch, “A 1280x720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86$\mu$m pixels, 1.066Geps readout, programmable event-rate controller and compressive data-formatting pipeline,” in _IEEE Intl. Solid-State Circuits Conf. (ISSCC)_ , 2020, pp. 112–114.
  * [3] G. Chen, H. Cao, J. Conradt, H. Tang, F. Rohrbein, and A. Knoll, “Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception,” _IEEE Signal Process. Mag._ , vol. 37, no. 4, pp. 34–49, 2020.
  * [4] P. K. Murali, M. Kaboli, and R. Dahiya, “Intelligent in-vehicle interaction technologies,” _Adv. Intell. Syst._ , vol. 4, no. 2, p. 2100122, 2022.
  * [5] W. Wang, E. Covi, A. Milozzi, M. Farronato, S. Ricci, C. Sbandati, G. Pedretti, and D. Ielmini, “Neuromorphic motion detection and orientation selectivity by volatile resistive switching memories,” _Adv. Intell. Syst._ , vol. 3, no. 4, p. 2000224, 2021.
  * [6] S. Zhong, Y. Zhang, H. Zheng, F. Yu, and R. Zhao, “Spike-based spatiotemporal processing enabled by oscillation neuron for energy-efficient artificial sensory systems,” _Adv. Intell. Syst._ , p. 2200076, 2022.
  * [7] H. Kim, S. Leutenegger, and A. J. Davison, “Real-time 3D reconstruction and 6-DoF tracking with an event camera,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2016, pp. 349–364.
  * [8] H. Rebecq, T. Horstschäfer, G. Gallego, and D. Scaramuzza, “EVO: A geometric approach to event-based 6-DOF parallel tracking and mapping in real-time,” _IEEE Robot. Autom. Lett._ , vol. 2, no. 2, pp. 593–600, 2017\.
  * [9] A. Z. Zhu, N. Atanasov, and K. Daniilidis, “Event-based visual inertial odometry,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2017, pp. 5816–5824.
  * [10] Y. Zhou, G. Gallego, and S. Shen, “Event-based stereo visual odometry,” _IEEE Trans. Robot._ , vol. 37, no. 5, pp. 1433–1450, 2021.
  * [11] A. Rosinol Vidal, H. Rebecq, T. Horstschaefer, and D. Scaramuzza, “Ultimate SLAM? combining events, images, and IMU for robust visual SLAM in HDR and high speed scenarios,” _IEEE Robot. Autom. Lett._ , vol. 3, no. 2, pp. 994–1001, Apr. 2018.
  * [12] H. Rebecq, G. Gallego, E. Mueggler, and D. Scaramuzza, “EMVS: Event-based multi-view stereo—3D reconstruction with an event camera in real-time,” _Int. J. Comput. Vis._ , vol. 126, no. 12, pp. 1394–1414, Dec. 2018.
  * [13] R. Benosman, C. Clercq, X. Lagorce, S.-H. Ieng, and C. Bartolozzi, “Event-based visual flow,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 25, no. 2, pp. 407–417, 2014.
  * [14] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “Unsupervised event-based learning of optical flow, depth, and egomotion,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2019, pp. 989–997.
  * [15] J. J. Hagenaars, F. Paredes-Valles, and G. C. H. E. de Croon, “Self-supervised learning of event-based optical flow with spiking neural networks,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 34, 2021, pp. 7167–7179.
  * [16] S. Shiba, Y. Aoki, and G. Gallego, “Secrets of event-based optical flow,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2022, pp. 628–645.
  * [17] C. E. Willert and J. Klinner, “Event-based imaging velocimetry: an assessment of event-based cameras for the measurement of fluid flows,” _Exp. Fluids_ , vol. 63, no. 6, pp. 1–20, 2022.
  * [18] T. Stoffregen, G. Gallego, T. Drummond, L. Kleeman, and D. Scaramuzza, “Event-based motion segmentation by motion compensation,” in _Int. Conf. Comput. Vis. (ICCV)_ , 2019, pp. 7243–7252.
  * [19] Y. Zhou, G. Gallego, X. Lu, S. Liu, and S. Shen, “Event-based motion segmentation with spatio-temporal graph cuts,” _IEEE Trans. Neural Netw. Learn. Syst._ , pp. 1–13, 2021.
  * [20] D. Falanga, K. Kleber, and D. Scaramuzza, “Dynamic obstacle avoidance for quadrotors with event cameras,” _Science Robotics_ , vol. 5, no. 40, p. eaaz9712, Mar. 2020.
  * [21] G. Cohen, S. Afshar, B. Morreale, T. Bessell, A. Wabnitz, M. Rutten, and A. van Schaik, “Event-based sensing for space situational awareness,” _J. Astronaut. Sci._ , vol. 66, no. 2, pp. 125–141, 2019.
  * [22] G. Gallego, H. Rebecq, and D. Scaramuzza, “A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2018, pp. 3867–3876.
  * [23] S. Shiba, Y. Aoki, and G. Gallego, “Event collapse in contrast maximization frameworks,” _Sensors_ , vol. 22, no. 14, pp. 1–20, 2022.
  * [24] U. M. Nunes and Y. Demiris, “Robust event-based vision model estimation by dispersion minimisation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2021\.
  * [25] G. Gallego, M. Gehrig, and D. Scaramuzza, “Focus is all you need: Loss functions for event-based vision,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2019, pp. 12 272–12 281.
  * [26] G. Gallego, T. Delbruck, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. Davison, J. Conradt, K. Daniilidis, and D. Scaramuzza, “Event-based vision: A survey,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 44, no. 1, pp. 154–180, 2022.
  * [27] C. Gu, E. Learned-Miller, D. Sheldon, G. Gallego, and P. Bideau, “The spatio-temporal Poisson point process: A simple model for the alignment of event camera data,” in _Int. Conf. Comput. Vis. (ICCV)_ , 2021, pp. 13 495–13 504.
  * [28] G. Gallego and D. Scaramuzza, “Accurate angular velocity estimation with an event camera,” _IEEE Robot. Autom. Lett._ , vol. 2, no. 2, pp. 632–639, 2017.
  * [29] H. Kim and H. J. Kim, “Real-time rotational motion estimation with contrast maximization over globally aligned events,” _IEEE Robot. Autom. Lett._ , vol. 6, no. 3, pp. 6016–6023, 2021.
  * [30] A. Z. Zhu, N. Atanasov, and K. Daniilidis, “Event-based feature tracking with probabilistic data association,” in _IEEE Int. Conf. Robot. Autom. (ICRA)_ , 2017, pp. 4465–4470.
  * [31] H. Seok and J. Lim, “Robust feature tracking in dvs event stream using Bezier mapping,” in _IEEE Winter Conf. Appl. Comput. Vis. (WACV)_ , 2020, pp. 1647–1656.
  * [32] T. Stoffregen and L. Kleeman, “Event cameras, contrast maximization and reward functions: an analysis,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2019, pp. 12 292–12 300.
  * [33] X. Peng, L. Gao, Y. Wang, and L. Kneip, “Globally-optimal contrast maximisation for event cameras,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 44, no. 7, pp. 3479–3495, 2022.
  * [34] A. Mitrokhin, C. Fermuller, C. Parameshwara, and Y. Aloimonos, “Event-based moving object detection and tracking,” in _IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS)_ , 2018, pp. 1–9.
  * [35] C. M. Parameshwara, N. J. Sanket, C. D. Singh, C. Fermüller, and Y. Aloimonos, “0-MMS: Zero-shot multi-motion segmentation with a monocular event camera,” in _IEEE Int. Conf. Robot. Autom. (ICRA)_ , 2021, pp. 9594–9600.
  * [36] S. Ghosh and G. Gallego, “Multi-event-camera depth estimation and outlier rejection by refocused events fusion,” _Adv. Intell. Syst._ , 2022.
  * [37] P. Charbonnier, L. Blanc-Feraud, G. Aubert, and M. Barlaud, “Deterministic edge-preserving regularization in computed imaging,” _IEEE Trans. Image Process._ , vol. 6, no. 2, pp. 298–311, 1997.
  * [38] T. Ozawa, Y. Sekikawa, and H. Saito, “Accuracy and speed improvement of event camera motion estimation using a bird’s-eye view transformation,” _Sensors_ , vol. 22, no. 3, 2022.
  * [39] T. D. Barfoot, _State Estimation for Robotics - A Matrix Lie Group Approach_. Cambridge University Press, 2015\.
  * [40] G. Gallego and A. Yezzi, “A compact formula for the derivative of a 3-D rotation in exponential coordinates,” _J. Math. Imaging Vis._ , vol. 51, no. 3, pp. 378–384, Aug. 2014.
  * [41] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “EV-FlowNet: Self-supervised optical flow estimation for event-based cameras,” in _Robotics: Science and Systems (RSS)_ , 2018, pp. 1–9.
  * [42] M. Gehrig, W. Aarents, D. Gehrig, and D. Scaramuzza, “DSEC: A stereo event camera dataset for driving scenarios,” _IEEE Robot. Autom. Lett._ , vol. 6, no. 3, pp. 4947–4954, 2021.
  * [43] A. Z. Zhu, D. Thakur, T. Ozaslan, B. Pfrommer, V. Kumar, and K. Daniilidis, “The multivehicle stereo event camera dataset: An event camera dataset for 3D perception,” _IEEE Robot. Autom. Lett._ , vol. 3, no. 3, pp. 2032–2039, Jul. 2018.
  * [44] M. Gehrig, M. Millhäusler, D. Gehrig, and D. Scaramuzza, “E-RAFT: Dense optical flow from event cameras,” in _Int. Conf. 3D Vision (3DV)_ , 2021, pp. 197–206.
  * [45] G. Taverni, D. P. Moeys, C. Li, C. Cavaco, V. Motsnyi, D. S. S. Bello, and T. Delbruck, “Front and back illuminated Dynamic and Active Pixel Vision Sensors comparison,” _IEEE Trans. Circuits Syst. II_ , vol. 65, no. 5, pp. 677–681, 2018.
  * [46] E. Mueggler, H. Rebecq, G. Gallego, T. Delbruck, and D. Scaramuzza, “The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM,” _Int. J. Robot. Research_ , vol. 36, no. 2, pp. 142–149, 2017.
  * [47] E. Mueggler, G. Gallego, H. Rebecq, and D. Scaramuzza, “Continuous-time visual-inertial odometry for event cameras,” _IEEE Trans. Robot._ , vol. 34, no. 6, pp. 1425–1440, Dec. 2018.
  * [48] C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck, “A 240x180 130dB 3$\mu$s latency global shutter spatiotemporal vision sensor,” _IEEE J. Solid-State Circuits_ , vol. 49, no. 10, pp. 2333–2341, 2014.
  * [49] T. Stoffregen, C. Scheerlinck, D. Scaramuzza, T. Drummond, N. Barnes, L. Kleeman, and R. Mahony, “Reducing the sim-to-real gap for event cameras,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2020, pp. 534–549.
  * [50] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 24, 2011, pp. 2546–2554.
  * [51] P. Corke, _Robotics, Vision and Control: Fundamental Algorithms in MATLAB_ , ser. Springer Tracts in Advanced Robotics. Springer, 2017.
  * [52] X. Clady, C. Clercq, S.-H. Ieng, F. Houseini, M. Randazzo, L. Natale, C. Bartolozzi, and R. Benosman, “Asynchronous visual event-based time-to-contact,” _Front. Neurosci._ , vol. 8, no. 9, 2014.


* [125] Page, S.E. The difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies; Princeton University Press: Princeton, 2007.
  * [126] Fiechter, J.; Kornell, N. How the wisdom of crowds, and of the crowd within, are affected by expertise. Cogn. Res. 2021, 6, 5.
  * [127] Werner, G.; Mountcastle, V.B. The variability of central neural activity in a sensory system and its implications for the central reflection of sensory events. J. Neurophysiol. 1963, 26, 958–977.
  * [128] Arieli, A.; Sterkin, A.; Grinvald, A.; Aertsen, A. Dynamics of ongoing activity: Explanation of the large variability in evoked cortical responses. Science 1986, 273, 1868–1871.
  * [129] Gold, J.I.; Shadlen, M.N. Neural computations that underlie decisions about sensory stimuli. Trends Cognit. Sci. 2001, 5, 10–16.
  * [130] Glimcher, P.W. Indeterminacy in brain and behaviour. Annu. Rev. Psychol. 2005, 56, 25–56.
  * [131] Schumacher, J.F.; Thompson, S.K.; Olman, C.A. Contrast response functions for single Gabor patches: ROI-based analysis over-represents low-contrast patches for GE BOLD. Front. Syst. Neurosci. 2011, 5, 1–10.
  * [132] Shadlen, M.N.; Shohamy, D. Perspective decision making and sequential sampling from memory. Neuron 2016, 90, 927–939.
  * [133] Webb, R. The neural dynamics of stochastic choice. Manag. Sci. 2019, 65, 230–255.
  * [134] Kurtz-David, V.; Persitz, D.; Webb, R.; Levy, D.J. The neural computation of inconsistent choice behaviour. Nature Commun. 2019, 10, 1583.
  * [135] Woodford, M. Modeling imprecision in perception, valuation, and choice. Annu. Rev. Econ., 2020, 12, 579–601.
  * [136] Yukalov, V.I.; Sornette, D. Preference reversal in quantum decision theory. Front. Psychol. 2015, 6, 1538.
  * [137] Martin, E.D. The Behavior of Crowds: A Psychological Study; Harper: New York, 1920.
  * [138] Sherif, M. The Psychology of Social Norms; Harper Collins: New York, 1936.
  * [139] Smelser, N.J. Theory of Collective Behavior; Free Press: Glencoe, 1963.
  * [140] Merton, R.K. Social Theory and Social Structure; Free Press: New York, 1968.
  * [141] Turner, R.H.; Killian, L.M. Collective Behavior; Prentice-Hall: Englewood Cliffs, 1993.
  * [142] Hatfield, E.; Cacioppo, J.T.; Rapson, R.L. Emotional Contagion; Cambridge University: New York, 1994.
  * [143] Brunnermeier, M.K. Asset Pricing under Asymmetric Information: Bubbles, Crashes, Technical Analysis, and Herding; Oxford University: Oxford, 2001.
  * [144] Yukalov, V.I.; Yukalova, E.P.; Sornette, D. Information processing by networks of quantum decision makers. Physica A 2018, 492, 747–766.
  * [145] Yukalov, V.I.; Yukalova, E.P.; Sornette, D. Role of collective information in networks of quantum operating agents. Physica A 2022, 598, 127365.
  * [146] Yukalov, V.I.; Yukalova, E.P. Self-excited waves in complex social systems. Physica D 2022, 433, 133188.
  * [147] Kühberger, A.; Komunska, D.; Perner, J. The disjunction effect: Does it exist for two-step gambles? Org. Behav. Hum. Decis. Process. 2001, 85, 250–264.
  * [148] Charness, G.; Rabin, M. Understanding social preferences with simple tests. Quart. J. Econ. 2002, 117, 817–869.
  * [149] Blinder, A.; Morgan, J. Are two heads better than one? An experimental analysis of group versus individual decision-making. J. Money Credit Bank. 2005, 37, 789–811.
  * [150] Cooper, D.; Kagel, J. Are two heads better than one? Team versus individual play in signaling games. Amer. Econ. Rev. 2005, 95, 477–509.
  * [151] Sutter, M. Are four heads better than two? An experimental beauty-contest game with teams of different size. Econ. Lett. 2005, 88, 41–46.
  * [152] Tsiporkova, E.; Boeva, V. Multi-step ranking of alternatives in a multi-criteria and multi-expert decision making environment. Inform. Sci. 2006, 176, 2673–2697.
  * [153] Charness, G.; Karni, E.; Levin, D. Individual and group decision making under risk: An experimental study of bayesian updating and violations of first-order stochastic dominance. J. Risk Uncert. 2007, 35, 129–148.
  * [154] Charness, G.; Rigotti, L.; Rustichini, A. Individual behavior and group membership. Amer. Econ. Rev. 2007, 97, 1340–1352.
  * [155] Chen, Y.; Li, S. Group identity and social preferences. Amer. Econ. Rev. 2009, 99, 431–457.
  * [156] Liu, H.H.; Colman, A.M. Ambiguity aversion in the long run: Repeated decisions under risk and uncertainty. J. Econ. Psychol. 2009, 30, 277–284.
  * [157] Charness, G.; Karni, E.; Levin, D. On the conjunction fallacy in probability judgement: New experimental evidence regarding Linda. Games Econ. Behav. 2010, 68, 551–556.
  * [158] Yukalov, V.I.; Sornette, D. Role of information in decision making of social agents. Int. J. Inf. Technol. Decis. Mak. 2015, 14, 1129–1166.