11(2.5, 0.4)

This paper has been accepted for publication at the
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. ©IEEE

# Event-based Background-Oriented Schlieren

Shintaro Shiba1,2, Friedhelm Hamann1,3, Yoshimitsu Aoki2, and Guillermo
Gallego1,3,4 1 Technische Universität Berlin, Berlin, Germany. 2 Department of
Electronics and Electrical Engineering, Faculty of Science and Technology,
Keio University, Kanagawa, Japan.
E-mail<EMAIL_ADDRESS>3 Science of Intelligence Excellence Cluster, Berlin,
Germany. 4 Einstein Center Digital Future, Berlin, Germany. Preprint of IEEE
T-PAMI paper. DOI: 10.1109/TPAMI.2023.3328188 0000-0001-6053-2285
0009-0004-8828-6919 0000-0001-7361-0027 0000-0002-2672-9241

###### Abstract

Schlieren imaging is an optical technique to observe the flow of transparent
media, such as air or water, without any particle seeding. However,
conventional frame-based techniques require both high spatial and temporal
resolution cameras, which impose bright illumination and expensive computation
limitations. Event cameras offer potential advantages (high dynamic range,
high temporal resolution, and data efficiency) to overcome such limitations
due to their bio-inspired sensing principle. This paper presents a novel
technique for perceiving air convection using events and frames by providing
the first theoretical analysis that connects event data and schlieren. We
formulate the problem as a variational optimization one combining the
linearized event generation model with a physically-motivated parameterization
that estimates the temporal derivative of the air density. The experiments
with accurately aligned frame- and event camera data reveal that the proposed
method enables event cameras to obtain on par results with existing frame-
based optical flow techniques. Moreover, the proposed method works under dark
conditions where frame-based schlieren fails, and also enables slow-motion
analysis by leveraging the event camera’s advantages. Our work pioneers and
opens a new stack of event camera applications, as we publish the source code
as well as the first schlieren dataset with high-quality frame and event data.
https://github.com/tub-rip/event_based_bos.

###### Index Terms:

Event camera, Schlieren imaging, Background-oriented schlieren, Optical flow,
Low-level vision.

##  1 Introduction

Sensing the flow of transparent media, such as air or water, is important for
various applications from aerodynamics to gas leakage detection. Optical
imaging is a useful tool to examine such transparent media because it can
capture the media with high detail in space-time remotely. Among existing
methods, schlieren imaging is a simple but efficient optical tool for seeing
the “invisible” [1, 2]: inhomogeneities in transparent media that are not
necessarily perceived by the naked eyes. It requires simple recording
settings: lenses, cameras, and mirrors or background patterns to image how
light rays deviate due to refractive index variations in the media. While it
was initially conceived as a visualization technique, recent developments in
schlieren and shadowgraphy have extended the usage to velocimetry [2, 3].
However, it requires a high-speed camera with a large spatial resolution to
analyze the velocity of the flow, such as convection. This is not only a
constraint for real-world applications but also a limitation of the
methodology because: ($i$) achieving high shutter speeds requires unnaturally
bright illumination, which is not always practical, ($ii$) transmitting and
processing the large amount of redundant data acquired involves high
bandwidth, storage, and power-hungry components, and ($iii$) regardless of the
large power consumption, the trade-off between speed and spatial resolution
limits accuracy in estimating the flow velocity.

Figure 1: In background-oriented schlieren (BOS) imaging local density
gradient variations between a camera and a background pattern lead to tiny
perceived changes on the image plane. We present how to combine events and
frames to observe schlieren in the scene and how to leverage the advantages of
event cameras to visualize gas streams such as the human breath.

Event cameras [4, 5] are novel bio-inspired sensors that respond to pixel-wise
intensity changes, which are not always visible to conventional frame-based
cameras. They offer advantages such as high speed, high dynamic range (HDR),
low power consumption, and data efficiency (temporal redundancy suppression)
[6], which makes them potential candidates to overcome the limitations of
traditional (i.e., frame-based) schlieren techniques. However, despite these
potential capabilities, the application of event cameras to imaging
applications is yet to be explored and developed.

This paper presents a novel technique, event-based background-oriented
schlieren (BOS), for sensing air convection with event cameras and proposes a
novel method to estimate the temporal derivative of air density from events
and frames (Fig. 1). Throughout the paper, we tackle the following challenges
of event-based BOS: ($i$) Theory. There is no established mathematical theory
for event-based schlieren techniques. ($ii$) Data. Event cameras sense only
increments of schlieren as opposed to the larger differences with respect to a
reference in frame-based BOS. ($iii$) Methodology. The origin of events in BOS
(flickering because they happen only at the edges of the background pattern)
and large amounts of noise are novel and difficult for previous work in event-
based vision. ($iv$) Evaluation. The real-world ground truth of the air
density is not easy to obtain, hence we need some proxy ground truth and
baselines.

First, we develop a theoretical connection between the schlieren and events,
showing that event cameras can sense the inhomogeneities of transparent media
in a more direct way (as flickering events) compared to frame-based cameras.
Such direct sensing of schlieren through event data enables us to observe air
convection at high speed more precisely and under challenging lighting
conditions. Second, we propose a novel method that extends the linearized
event generation model with physically-inspired parameterization to estimate
the temporal density fluctuation due to the schlieren. Third, in order to
evaluate the estimated density change, whose real-world ground truth is not
easy to obtain, we establish the evaluation method using optical flow, by
revealing the theoretical connection between the temporal density change and
optical flow (i.e., pixel displacement). Using a co-located frame-based camera
enables us to benchmark different methods of estimating temporal density
change as a computer vision problem. The experimental results show that: ($i$)
our proposed method recovers the flow that corresponds to the temporal change
of density gradient by comparing with the standard frame-based methods and
other baseline methods, ($ii$) flickering-like events are a more direct
measurement of such schlieren, ($iii$) event cameras record the density
inhomogeneities even in poor lighting conditions, which state-of-the-art
frame-based algorithms cannot provide, and ($iv$) the high temporal resolution
of event cameras enables slow-motion schlieren analysis.

The main technical contributions of this work are:

  * •

A novel method for computation of schlieren combining events and frames (Secs.
3 and 4). The proposed method is rigorously obtained and well connected with
the physical model of the sensors involved via the linearized event generation
model.

  * •

The first schlieren event-frames dataset (Sec. 5). We publicly provide
recordings of several schlieren scenes by means of events and frames, at high
resolution (1 Mpixel), accurately synchronized and calibrated using an in-
house acquisition system.

  * •

A thorough comparison with baseline methods despite the lack of truly ground
truth data in this type of turbulent fluid dynamics phenomena (Sec. 6).

  * •

A simulation experiment based on the high-speed frame data from [3] to confirm
the results of event-based BOS using those from frame-based BOS (Sec. 7).

To the best of our knowledge, this is the first work showing the potential
advantages of event cameras for schlieren imaging applications.

##  2 Related Work

###  2.1 Background-Oriented Schlieren

Schlieren photography was invented in 1864 to study the flow of air around
objects moving at supersonic speed [1]. In contrast to other imaging and
velocimetry techniques such as particle image velocimetry [7, 8], it does not
require any particle seeding in the media of interest. Among different
schlieren-imaging techniques (see Tab. I), BOS is a relatively recent
technique since it utilizes digital image processing [9]. In BOS (Fig. 2), an
object of interest with density variations (e.g., the hot air stream from a
burning candle) is placed between the camera and a constant (non-moving)
background pattern. The schlieren generates complex deformation to the
background pattern, which is observed by cameras as the apparent motion of the
background pattern with respect to a reference image (without density
variations) [9]. Different methods have been proposed to compute the
displacement vector field of the apparent motion, such as using cross-
correlation [10], optical flow [11], or wavelet-based analysis [12]. As
equally important as the data processing method is the data acquisition setup.
Best practices for parameter settings, such as the distance from the camera to
the background and the media, are provided in [2].

TABLE I: Comparison of various schlieren imaging techniques and the physical
quantities they measure.

Method Observation Shadowgraphy [13, 14, 15]
$\displaystyle\frac{\partial^{2}\rho}{\partial\mathbf{x}^{2}}$ Toepler’s
schlieren photography [16, 14]
$\displaystyle\frac{\partial\rho}{\partial\mathbf{x}}$ Laser speckle
photography [14] $\displaystyle\frac{\partial\rho}{\partial\mathbf{x}}$ Frame-
based BOS [14] $\displaystyle\frac{\partial\rho}{\partial\mathbf{x}}$ Event-
based BOS (this work) $\displaystyle\frac{\partial^{2}\rho}{\partial
t\partial\mathbf{x}}$

Figure 2: Background-Oriented Schlieren (BOS) setup.

BOS has been used to image various transparent media, such as shock waves from
explosions [17], turbulent flows [3], and shock waves underwater [18]. Also,
the background pattern of BOS can be extended to natural images [19], which
allows us to image the flow with a large field-of-view (FOV). In [20], BOS is
utilized to visualize supersonic jets in flight, by leveraging the natural
vegetation of the terrain seen from above as the constant background pattern.
The large FOV is one of the unique characteristics of BOS, unlike other
schlieren techniques, which enables measuring natural outdoor scenes [2].
Notwithstanding, BOS can be used as input to other analysis tools, such as
Dynamic Mode Decomposition (DMD) to reveal the main frequency modes of
variation of the signal in space and time [21], which ultimately inform about
the physical parameters of the turbulent flow. Recently, some works have
extended BOS from an imaging technique to a quantitative method, e.g., [22]
measures density of axisymmetric supersonic flow. In [3], a method is proposed
to extract velocity data from flows. For this application, Kymography works
better than classical image correlation, and the self-similarity of round
turbulent jet velocity appears in the schlieren results.

###  2.2 Event Cameras

Event cameras are a relatively new technology compared to BOS imaging with
standard frame-based cameras. Since the 2008 seminal work [4], they have been
slowly commercialized and explored in computer vision and robotics for various
applications. Event cameras naturally respond to motion in the scene at high
speed and HDR in a data-efficient manner, hence large progress has been made
in motion-related tasks, such as optical flow estimation [23, 24, 25, 26, 27],
ego-motion estimation [28, 29, 30, 31], SLAM [32, 33, 34, 35], or video
deblurring and frame interpolation [36, 37, 38].

Only recently, the larger spatial resolution of event cameras and higher fill
factor of their pixels [5, 39] has enabled fine-detail applications that were
not possible with older models. Some works have explored event cameras for
detecting small changes in the scene. These include vibration monitoring [40],
particle-image velocimetry [41], and time-resolved 3D fluid flow
reconstruction via collimated illumination [42]. These works open another
stack of event camera applications in the field of fluid dynamics. Event-based
BOS aims at pushing the limits, by imaging and quantifying flow fields without
any particle seeding. Event cameras are available from several manufacturers,
costing in the range of 2000 to 6000 USD (as of 2023). We refer to [6, 43] for
more details on the different camera types and manufacturers.

Physics-based Methods. The method developed in this work is related to a body
of literature that leverages the physics of the camera (event generation
model) in an optimization framework to either estimate some motion parameters
(e.g., feature tracks [44], camera ego-motion [45, 46, 35], optical flow with
additionally-provided frame information [47]), and/or a grayscale image given
the motion [48, 49, 47]. Our work builds on top of the event generation model
of the camera [6], extending it to the considered BOS problem. Specifically,
we extend [44], which was designed for sparse patches around keypoints
undergoing Euclidean in-plane motion, to the case of dense flow fields caused
by complex (i.e., non-rigid) schlieren.

##  3 Event-based Schlieren

###  3.1 Principles of Frame-based BOS

In frame-based BOS the schlieren object $S$ (e.g., a gas with varying density)
produces an apparent displacement of the background pattern, which is measured
with respect to the initial state (i.e., image acquired in the absence of
density gradient). The displacement $\Delta\mathbf{x}\doteq(\Delta x,\Delta
y)^{\top}$ is directly related to the small deflection angle
$\boldsymbol{\epsilon}\doteq(\epsilon_{x},\epsilon_{y})^{\top}$ (Fig. 2) via
the distance from lens to $S$ ($Z_{A}$), the distance from $S$ to the
background ($Z_{D}$), and the focal length of the lens $f$ [14]:

$\Delta\mathbf{x}\approx
f\left(\frac{Z_{D}}{Z_{D}+Z_{A}-f}\right)\boldsymbol{\epsilon}.$ (1)

On the other hand, for the refractive index $n$, the angle
$\boldsymbol{\epsilon}$ is the result of aggregating the spatial gradient
$\partial{n}/\partial{\mathbf{x}}$ along the length $Z$ of the schlieren
object $S$ on the optical axis:

$\boldsymbol{\epsilon}=\frac{1}{n}\int\frac{\partial
n}{\partial\mathbf{x}}dz=\frac{Z}{n_{\infty}}\frac{\partial
n}{\partial\mathbf{x}},$ (2)

where the ambient-air refractive index is given as $n_{\infty}$. Finally, $n$
is related to the density $\rho$ of the gas (schlieren object) via the
Gladstone-Dale relation, $n=G\rho+1$, with constant $G=2.23\times
10^{-4}\text{m}^{3}/\text{kg}$ [14].

In short, the spatial gradient of the density
$\partial{\rho}/\partial{\mathbf{x}}$ within a gas causing schlieren can be
directly quantified by measuring the pixel displacement $\Delta\mathbf{x}$:

$\Delta\mathbf{x}\propto\frac{\partial\rho}{\partial\mathbf{x}},$ (3)

as summarized in Tab. I. Here, the displacement is measured against the
initial state (the background pattern), hence the corresponding density-
gradient field is the change with respect to the initial (also called
“reference”) state.

###  3.2 Principles of Event-based BOS

Figure 3: Frame-based BOS and event-based BOS.

One of the main differences between frame-based BOS and event-based BOS is
that event cameras only sense temporal changes of the scene, while the former
measures the displacement between a reference frame and the current frame
(Fig. 3). Hence, the key challenge is how we can relate events (the
asynchronous intensity changes between two timestamps $t_{1}$ and $t_{2}$) to
the density $\rho$. Since events are very noisy [50, 6], accumulating the
differences between far away timestamps to estimate the same displacement as
frame-based BOS (3) leads to high noise levels [51, 52], which makes it
difficult to estimate this displacement with events.

In order to establish the theoretical connection between schlieren and events,
let us first extend the previous frame-based BOS theory to compute the
displacement between two _nearby_ timestamps. Given frames at timestamps
$t_{1},t_{2}$, their displacements from a reference frame at $t_{\text{ref}}$
(1) are $\Delta\mathbf{x}(t_{\text{ref}},t_{1})$ and
$\Delta\mathbf{x}(t_{\text{ref}},t_{2})$. The optical flow
$\mathbf{v}(\mathbf{x})=\partial{\mathbf{x}}/\partial{t}$ between consecutive
frames for small $\Delta t=t_{2}-t_{1}$ is

$\mathbf{v}(\mathbf{x})=\frac{\Delta\mathbf{x}(t_{\text{ref}},t_{2})-\Delta\mathbf{x}(t_{\text{ref}},t_{1})}{\Delta
t}.$ (4)

From the frame-based BOS theory, the displacement at each timestamp can be
related to the density gradient as follows (3):

$\begin{split}\Delta\mathbf{x}(t_{\text{ref}},t_{1})\propto\frac{\partial\rho_{t1}}{\partial\mathbf{x}},\\\
\Delta\mathbf{x}(t_{\text{ref}},t_{2})\propto\frac{\partial\rho_{t2}}{\partial\mathbf{x}}.\end{split}$
(5)

Plugging (5) into (4), using finite-difference approximations and Schwarz’s
theorem, gives:

$\begin{split}\mathbf{v}(\mathbf{x})&\propto\frac{1}{\Delta
t}\bigl{(}\frac{\partial\rho_{t2}}{\partial\mathbf{x}}-\frac{\partial\rho_{t1}}{\partial\mathbf{x}}\bigr{)}\\\
&=\frac{1}{\Delta
t}\frac{\partial}{\partial\mathbf{x}}\bigl{(}\rho_{t2}-\rho_{t1}\bigr{)}\\\
&\approx\frac{\partial}{\partial\mathbf{x}}\frac{\partial}{\partial t}\rho,\\\
&=\frac{\partial}{\partial
t}\frac{\partial}{\partial\mathbf{x}}\rho.\quad\text{(Schwarz's
thm)}\end{split}$ (6)

That is, the optical flow between two nearby timestamps is related to the
_temporal derivative of the density gradient_ (see the last row of Tab. I).
Since events are the measurements between such nearby timestamps, the key
question is how can optical flow (i.e., spatio-temporal derivative of the
density) be estimated from event data.

(a) Recording system

(b) Events and frame

Figure 4: (a) Actual synchronized data recording system, combining an event
camera and a frame-based camera via a beamsplitter (Sec. 5.1). (b) Data:
events (red and blue, colored according to polarity) during a short time
window overlaid on a grayscale frame (a $100\times 100$ pixel region for
better visualization).

##  4 Estimation Method

One of the main challenges of event-based BOS is its data modality: events
generated by schlieren objects are sparse, happening only at the edges of the
background pattern (Fig. 4(b)) and in a flickering form. Previous event-based
optical flow estimation methods [26, 24, 25] often assume a continuous, non-
flickering apparent motion of the visual patterns on the image plane. Also,
events triggered during the short time interval needed to capture fine details
of the complex motion patterns are few compared to those in scenes from
typical optical flow benchmarks [53, 54]. Consequently prior methods fail to
produce accurate flow since they are not tailored to the schlieren scenario,
as we show in Sec. 6.2. Due to these challenges, we propose a method that
combines events and knowledge of the background pattern (e.g., frames) to
estimate the flow. The proposed method extends the linearized event generation
model (LEGM) [44, 46, 47, 35] to the characteristics of our problem. The
overall pipeline is described in Fig. 5.

Figure 5: Block diagram of the objective $E_{\text{data}}$ in (11). On the top
branch, events are integrated in time using (8) and smoothed with a Gaussian
kernel ($\sigma=2$ px) to produce the measured brightness increment image
$\Delta L$. The bottom branch shows how to compute the predicted brightness
increment $\Delta\hat{L}$ from the frame and the problem unknowns: the
translation field $\mathbf{p}$ and the Poisson parameters of the flow,
$\mathbf{q}$. The flow $\mathbf{v}$ and $\mathbf{p}$ are pseudo-colored (see
color wheel). Same data as Fig. 4.

###  4.1 Event Generation Model

An event $e_{k}\doteq(\mathbf{x}_{k},t_{k},p_{k})$ conveys that the
logarithmic brightness $L$ at pixel $\mathbf{x}_{k}$ changes by a specified
contrast sensitivity $C$ [4, 6]:

$\Delta L(\mathbf{x}_{k},t_{k})\doteq
L(\mathbf{x}_{k},t_{k})-L(\mathbf{x}_{k},t_{k}-\Delta t_{k})=p_{k}\,C,$ (7)

where polarity $p_{k}\in\\{+1,-1\\}$ is the sign of the brightness change, and
$\Delta t_{k}$ is the time since the last event at pixel $\mathbf{x}_{k}$.
Given a set of events $\mathcal{E}\doteq\\{e_{k}\\}_{k=1}^{N_{e}}$, summing
their polarities pixelwise produces a brightness increment image:

$\Delta L(\mathbf{x})=\sum_{k}p_{k}C\,\delta(\mathbf{x}-\mathbf{x}_{k}),$ (8)

where the Kronecker $\delta$ selects the pixel $\mathbf{x}_{k}$. The LEGM
states that, assuming brightness constancy during a small $\Delta
t=t_{N_{e}}-t_{1}$, the increment (7) is caused by brightness gradients
$\nabla L$ moving with image velocity $\mathbf{v}$ [45]:

$\Delta L(\mathbf{x})\approx-\nabla L(\mathbf{x})\cdot\Delta\mathbf{x}=-\nabla
L(\mathbf{x})\cdot\mathbf{v}(\mathbf{x})\Delta t.$ (9)

###  4.2 Optimization Objective

We cast the problem of estimating the displacement (6) as an optimization one,
where we minimize the mismatch between the event data (in the form of (8)) and
its prediction $\Delta\hat{L}$ via (9) exploiting the knowledge of the
background pattern from a frame $\hat{L}$. This idea is summarized in Fig. 5.

To allow for the fact that $\hat{L}$ may not be perfectly aligned with the
corresponding events, we augment the model (9) with a translation warp
$\hat{L}(\mathbf{W}(\mathbf{x};\mathbf{p}))$, where
$\mathbf{W}(\mathbf{x};\mathbf{p})=\mathbf{x}+\mathbf{p}$, and $\mathbf{p}$
denotes a small per-pixel translation.

Our composite objective (i.e., loss) function implies a joint optimization
over the flow and alignment parameters:

$E(\mathbf{v},\mathbf{p})\doteq
E_{\text{data}}(\mathbf{v},\mathbf{p};\mathcal{E})+E_{\text{reg}}(\mathbf{v},\mathbf{p};\mathcal{E}).$
(10)

The data-fidelity term measures the goodness of fit between the event data
$\mathcal{E}$ and its prediction with our model:

$E_{\text{data}}\doteq\left\|\frac{\Delta\hat{L}}{\|\Delta\hat{L}\|_{2}}(\mathbf{x})-\frac{\Delta
L}{\|\Delta L\|_{2}}(\mathbf{x})\right\|_{\gamma},$ (11)

where $\gamma$ is the $L^{1}$ norm (robust norm). Since $C$ in (8) is unknown,
we compute the difference between normalized brightness increments (norms are
over the pixel domain $\Omega$).

The regularizer penalizes the non-smoothness of the flow $\mathbf{v}$ and the
magnitude of the per-pixel translation $\mathbf{p}$:

$E_{\text{reg}}\doteq\lambda_{1}\|w(\mathbf{x})\,\nabla\mathbf{v}(\mathbf{q}(\mathbf{x}))\|_{1}+\lambda_{2}\|\mathbf{p}(\mathbf{x})\|_{1}.$
(12)

The flow regularizer (first term in (12)) is explained in Sec. 4.4, after the
flow parameterization is introduced. For the second term, the magnitude of
$\mathbf{p}$ is given by its $L^{1}$ norm over the pixel domain. In the
experiments, we set $\lambda_{1}=0.5$ and $\lambda_{2}=0.1$.

###  4.3 Physically-motivated Parameterization

Swapping the mixed derivatives (Schwarz’s theorem) in (6), the flow
$\mathbf{v}\sim\frac{\partial}{\partial\mathbf{x}}\frac{\partial\rho}{\partial
t}$ is interpreted as the spatial gradient of $\frac{\partial\rho}{\partial
t}$. Thus (6) admits two interpretations. ($i$) from left to right: once
estimated, the flow may be Poisson-integrated [55] to obtain
$\frac{\partial\rho}{\partial t}$, (as the best $L^{2}$ fit to the estimated
flow [56, 49]). ($ii$) from right to left: the flow may be obtained as the
spatial (e.g., Sobel) gradient of a scalar field $\frac{\partial\rho}{\partial
t}$. In contrast to most optical flow estimation methods, which parametrize
$\mathbf{v}(\mathbf{x})$ directly in terms of its $x$ and $y$ components, we
go one step further and exploit the above second interpretation of (6) to
parametrize the flow by means of $\mathbf{q}\equiv\frac{\partial\rho}{\partial
t}$, which we call the Poisson parameters of the flow. This not only reduces
the complexity of the problem (number of variables being optimized), thus
conferring robustness, but also provides a strong link with the physical
meaning of the variables: according to (6), the resulting flow actually
represents the schlieren objects. Figure 6 shows examples of the Poisson
parameters $\mathbf{q}$.

Figure 5 summarizes the visual quantities involved in the calculation of (11).
The candidate scalar parameter field $\mathbf{q}$ is converted (via Sobel
operator) into the vector flow field $\mathbf{v}$. The flow $\mathbf{v}$ and
translation field $\mathbf{p}$ are used in the augmented model of (9) to
generate a predicted (i.e., modeled) brightness increment image. On the other
hand, events $\mathcal{E}$ are summed in (8) and Gaussian-smoothed to produce
a measured brightness increment image. The difference between the measured and
predicted brightness increments provides an error signal that is used to drive
the iterative refinement of the unknown variables $\mathbf{p}$ and
$\mathbf{q}$.

###  4.4 Flow Regularizer

We penalize the non-smoothness of the flow using a weighted Total Variation
(TV) (see (12)). As illustrated in Fig. 4(b), it is difficult to estimate
accurate flow in regions with very few events, which correspond to constant
(e.g., zero) flow, hence we impose this prior knowledge as a regularizer to
encourage zero flow therein. Specifically, from the events $\mathcal{E}$ we
compute a Gaussian-smoothed histogram
$h(\mathbf{x};\mathcal{E})=\sum_{k}\mathcal{N}(\mathbf{x};\mathbf{x}_{k},\sigma^{2})$
(with $\sigma=5$ px) and normalize it to the range $[0,1]$. Then, we define
weight function $w(\mathbf{x})\doteq
1-\alpha/h(\mathbf{x};\mathcal{E})(\mathbf{x})$ (large in ill-posed regions
with very few events), with $\alpha=0.95$ in the experiments.

###  4.5 Optimization

Multi-scale. For improved convergence, a coarse-to-fine patch-based approach
is used for $\mathbf{v},\mathbf{p}$ and the loss function (10). The coarsest
patch size is $64\times 64$ px and we use four resolution levels in a
pyramidal fashion, resulting in finest patches of $8\times 8$ px. To reach
pixel density from the finest patches, we use bilinear interpolation.

Implementation. We use events in the fixed time interval (i.e., $120$ fps) for
optimization across all sequences. As an optimizer, we use Adam [57] with 600
iterations. The learning rate is set to 0.05, with the decay of 0.1. The
initialization of the first frame at the coarsest scale is: zero for
$\mathbf{p}$ and $\mathbf{v}$ (when applicable) and random in $[-1,1]$ for the
Poisson parameters $\mathbf{q}$. We found the latter to be better than also
setting $\mathbf{q}$ to zero. Then, the initialization of the next levels uses
the optimization results from the previous scales (i.e., coarse-to-fine
approach).

(a) Poisson images ($\mathbf{q}$)

(b) Optical flow ($\mathbf{v}$)

Figure 6: Example of Poisson parameters and optical flow.

##  5 Physical Setup and Data

###  5.1 Recording Setup

Co-capture System. To achieve high-quality recordings of frames and events, we
build our own acquisition system. Although some devices exist that record
colocated events and frames (such as DAVIS [58, 59]), their data quality
(resolution, dynamic range, etc.) is limited and not suitable for BOS
applications. Our custom-built co-capture system consists of a frame camera
(Basler acA1300-200um, 1280$\times$1024 px) and the latest generation event
camera (Prophesee EVK3 Gen4, 1280$\times$720 px [5]), sharing the same optical
axis by using a beamsplitter (Plate Bs C-Mount VIS50R/50T). Both cameras are
hardware-triggered for accurate synchronization and are calibrated to achieve
accurate pixel alignment, following [60]. Figure 4 shows the camera system and
an example of acquired data. Further details about the used recording system
can be found in [61].

Optical Setup. The field of view (FOV) of our cameras is limited by the
beamsplitter ($\approx 15^{\circ}$), hence we set the distance between the
cameras and the background to 3.3 m. We use randomly-generated background
patterns that cover the whole FOV, where black dots (covering approximately 2
to 3 px in the image plane) are printed on a white paper.

The data quality also depends on the distance between the camera and the
schlieren object. The schlieren are more visible (larger pixel displacement
$\Delta\mathbf{x}$) by keeping $Z_{A}$ small (object closer to the camera). At
the same time, the camera system has to be focused both on the background
pattern and the schlieren object, thus $Z_{A}$ cannot be too small. We
experimentally found distance $Z_{D}=1.6$ m to be a good compromise between
both opposing effects. To control the scene brightness and achieve uniform
illumination in the background, we use LED panels (four Eurolite LED PLL-360).
This illumination allows us to lower the aperture to an f-number of 10,
leading to a higher depth of field. Note that our beamsplitter setup leads to
a 50% split of the light reaching each camera of the acquisition system.

###  5.2 Data Acquired

We record multiple sequences with natural and non-natural (forced) air
convection, which are summarized in Tab. II. For natural convection, we use
heat sources, such as a hot plate, a hair dryer (switched off), and ice. To
demonstrate the HDR capabilities of event cameras, we record the data in ($i$)
bright conditions ($\approx$ 4000 $\mathrm{lx}$) and ($ii$) low-light
conditions ($\approx$ 225 $\mathrm{lx}$). The low-light condition is set to be
darker than normal office lighting, which is a more natural condition for
real-world applications.

TABLE II: Parameters of the recorded sequences.

Sequence Convection Luminance Duration Event rate [lx] [s] [Mev/s] Hot plate 1
Natural 4000 $19.390\,846$ $11.293\,378\,174\,423\,127$ Hot plate 2 Natural
225 $19.774\,873$ $5.081\,686\,390\,602\,862$ Hair dryer (OFF) 1 Natural 4000
$13.499\,232\,000\,000\,001$ $5.072\,274\,407\,907\,057$ Hair dryer (OFF) 2
Natural 4000 $19.719\,239$ $5.296\,867\,794\,948\,882$ Hair dryer (OFF) 3
Natural 225 $14.683\,165$ $2.753\,698\,742\,743\,815\,8$ Crushed ice Natural
4000 $17.375\,194$ $5.040\,164\,501\,184\,85$ Hair dryer (ON) Forced 4000
$13.387\,983$ $14.968\,827\,119\,066\,404$ Breathing 1 Forced 4000
$12.779\,248$ $4.001\,311\,188\,263\,973$ Breathing 2 Forced 4000
$13.043\,173$ $3.680\,195\,915\,518\,409\,4$ Helium (synthetic) [3] Forced –
$0.5$ $161.3$

Each sequence is approximately 10 to 20 seconds long and consists of events,
frames and a calibration parameter file. The recording starts with the scene
in the absence of the schlieren object, which is useful for frame-based BOS
methods (reference frame). All sequences are recorded at normal room
temperature ($\approx 24^{\circ}$C). For the forced convection sequence of the
running hair dryer, we set the event camera’s refractory period to its minimum
possible value to capture the fast dynamics of the airflow. In total, we
record nine sequences, each of which has up to 200M events. Regarding storage,
events take about 1/10 of the data size required to store frames (e.g., 800 MB
vs. 7.8 GB for a hot plate sequence).

Hot plate |  |
---|---|---
Hair dryer 1 |  |
Hair dryer 2 |  |
Breath 2 |  |
Hotplate (dark) |  |
|  (a) Frame data |  (b) GT (Flow from frames)
Figure 7: _Dataset: samples of frame and frame-based flow._ Frames mapped into
the event-camera image plane are shown on the left. The estimated optical flow
(inside the ROI) is shown on the right. For the low-light sequences, the
frame-based method fails to estimate reasonable flow. Nevertheless, we show
them for completeness.

Frames of sample sequences are shown in Fig. 7. Each frame is mapped from its
original resolution (1280$\times$1024 px) to the event-camera resolution
(1280$\times$720 px) (see Sec. 5.1).

Since we cannot obtain real ground truth (GT), we use frame-based estimated
flow as GT flow (Fig. 7). The calculation of the flow is based on the
classical Farnebäck algorithm [62] with four pyramidal scales at the frame
rate (120 fps). We test different parameters and find no significant
difference on the quality of the results. Before settling for Farnebäck’s
algorithm, we tested recent DNN-based state-of-the-art methods, such as [63,
64], and found that they do not produce reasonable flow. Figure 8 shows the
comparison of several frame-based optical flow estimation methods: two state-
of-the-art optical flow and video-frame interpolation works [63, 64] and
Farnebäck’s method. Due to the large gap between the training datasets of [63,
64] and our dataset, these recent DNN-based methods fail to estimate
reasonable flow. Farnebäck’s algorithm works robustly and better, because
($i$) the background pattern is parallel to the image plane, ($ii$) the scene
has no occlusions, ($iii$) the background pattern has clear and random edges
that are useful to calculate the deformation between two frames. Since we
cannot determine the real GT, we do not explore a further analysis of frame-
based estimation methods, which we leave for future research, such as
simulation. That is, to establish the first event-based BOS problem settings
we leverage the knowledge of established frame-based BOS techniques. Note that
the quantitative evaluation is only based on the well-illuminated sequences,
since the frame-based flow degrades in dark scenes (see Sec. 6.6). We publish
the dataset and the code to compute the GT.

Hot plate |  |  |
---|---|---|---
Hair Dryer |  |  |
Breath |  |  |
|  (a) Farnebäck’s [62] |  (b) FlowFormer [63] |  (c) RIFE [64]
Figure 8: Different frame-based optical flow methods.

##  6 Experimental Evaluation

This section reports the performance of the proposed estimation method and its
properties. First, we explain the baseline methods and evaluation metrics
(Sec. 6.1). Second, we benchmark the accuracy of all methods considered (Sec.
6.2). Third, we show the capabilities of our method in low-light conditions
(Sec. 6.3) and how it achieves high temporal resolution (1200 Hz “slow
motion”) in Sec. 6.4, including a velocimetry application (Sec. 6.5). Finally,
we analyze the proposed method further, especially regarding the dependency on
frames (Secs. 6.6 and 6.7), its sensitivity to hyper-parameters (Sec. 6.8),
and the effect of event warping (Sec. 6.9).

###  6.1 Evaluation Metrics and Baseline Methods

Evaluation Metrics. We evaluate the proposed method in terms of optical flow
$\mathbf{v}$ accuracy. Two variants of the method are assessed: ($i$) using
$\mathbf{q}$ as parameterization, from which we obtain $\mathbf{v}$ afterwards
via (6), and ($ii$) using $\mathbf{v}$ directly.

The optical flow evaluation metrics are the average endpoint error (AEE), the
percentage of pixels with AEE $>1$ px (denoted by “% Out”), and the angular
error (AE). We select the time interval (from 1 to 4 s) and region of interest
(ROI) to remove objects, such as a hair dryer and a face from the scene. All
metrics are computed over pixels with at least one event inside the ROI.

Table III reports the detailed duration, ROI, and total number of events used
for the benchmark. The duration is selected such that the quality of schlieren
is the best and stable. For the “Hair dryer (ON)” sequence, we limit the
height of the ROI due to extremely large number of events observed: otherwise,
we set the ROI to have approximately 720$\times$512 px.

TABLE III: Details of the benchmark. “ROI position” contains the coordinates
of the top-left corner.

Sequence ROI size [px] ROI position [px] Duration [s] Total events Hot plate 1
640$\times$720 [320, 0] 10 to 14 $51\,900\,802$ Hot plate 2 (dark)
640$\times$720 [420, 0] 12 to 14 $12\,912\,262$ Hair dryer (OFF) 1
640$\times$640 [320, 0] 4 to 7 $13\,498\,252$ Hair dryer (OFF) 2
512$\times$640 [384, 0] 6 to 7 $4\,089\,883$ Hair dryer (dark) 512$\times$640
[384, 0] 5 to 7 $3\,460\,579$ Crushed ice 512$\times$512 [384, 208] 8 to 11
$5\,856\,190$ Hair dryer (ON) 1280$\times$200 [0, 260] 3.3 to 4.3
$17\,860\,129$ Breathing 1 590$\times$600 [400, 0] 4.36 to 5.5 $2\,783\,122$
Breathing 2 640$\times$640 [447, 0] 2.5 to 3.5 $1\,811\,889$ Total – – 18.14
$114\,173\,108$

Baselines. As baseline flow estimators we use the two self-implemented methods
from events because, to the best of our knowledge, there are no methods that
estimate schlieren flow from event camera data.

  * •

The Multi-reference Contrast Maximization (MCM) [26] is a state-of-the-art
optical flow estimation algorithm from events alone. It is a model-based
method, hence there is no mismatch in the training dataset (due to our
specific background pattern). We use the events between two consecutive frames
(i.e., in a time span of 8.3 ms).

  * •

Flow estimation from reconstructed intensity images: we use E2VID [65] (a
learning-based approach) to compute grayscale images from events and then
apply the same (frame-based) optical estimator as the one for the GT. Images
are reconstructed at 120 fps, i.e., the same frequency as the frames.

To the best of our knowledge, we found no methods with publicly-available
implementation combining events and frames to estimate the optical flow, we
therefore believe this is a best-effort comparison. Also, notice that we do
not train a Deep Neural Network (DNN) model with the supervisory GT flow, as
the purpose of the paper is not a purely data-driven approach, but to develop
an interpretable model-based method, by deriving a connection between the
physical parameters and the data.

###  6.2 Optical Flow Evaluation

TABLE IV:  Results of optical flow estimation.

Hair dryer (OFF) 1 Hair dryer (OFF) 2 Hot plate 1 Hair dryer (ON) AEE
$\downarrow$ %Out $\downarrow$ AE $\downarrow$ AEE $\downarrow$ %Out
$\downarrow$ AE $\downarrow$ AEE $\downarrow$ %Out $\downarrow$ AE
$\downarrow$ AEE $\downarrow$ %Out $\downarrow$ AE $\downarrow$ MCM [26]
$1.425\,00$ $35.639\,15$ $0.621\,26$ $0.421\,394\,212\,6$ $10.886\,155\,26$
$0.476\,29$ $0.399\,829\,15$ $21.788\,710\,999\,951\,608$ $0.426\,22$
$0.287\,47$ $5.933\,31$ $0.711\,78$ E2VID [65] $1.054\,680\,703\,374\,11$
$39.067\,61$ $0.677\,31$ $1.090\,89$ $37.733\,53$ $0.670\,48$ $1.091\,72$
$32.121\,34$ $0.611\,25$ $0.810\,97$ $25.997\,47$ $0.587\,48$ Ours (Flow)
$0.674\,770\,877\,541\,533$ $22.103\,63$ $0.403\,57$ $0.688\,19$ $24.930\,27$
$0.447\,83$ $0.809\,93$ $30.289\,05$ $0.544\,33$ $0.309\,57$ $6.756\,17$
$0.258\,33$ Ours (Poisson) $0.383\,128\,899\,671\,618$ $9.319\,32$ $0.299\,45$
$0.395\,36$ $10.173\,93$ $0.336\,96$ $0.486\,88$ $12.214\,58$ $0.420\,67$
$0.215\,21$ $0.924\,24$ $0.201\,59$ Crushed ice Breathing 1 Breathing 2 MCM
[26] $1.089\,83$ $96.964\,14$ $0.823\,28$ $1.769\,473\,693\,921\,366\,9$
$49.552\,307\,793\,363\,9$ $0.853\,121\,207\,924\,258\,5$ $2.055\,86$
$78.690\,12$ $0.973\,42$ E2VID [65] $1.249\,44$ $55.029\,76$ $0.790\,73$
$1.013\,65$ $42.071\,63$ $0.692\,37$ $1.055\,68$ $43.348\,11$ $0.699\,08$ Ours
(Flow) $0.586\,75$ $21.814\,87$ $0.451\,84$ $0.664\,75$ $11.871\,93$
$0.341\,02$ $0.556\,92$ $17.715\,66$ $0.438\,14$ Ours (Poisson) $0.325\,60$
$5.176\,59$ $0.301\,11$ $0.345\,07$ $6.321\,94$ $0.203\,33$ $0.476\,23$
$8.028\,23$ $0.410\,18$

Hot plate |  |  |  |  |
---|---|---|---|---|---
Hot plate |  |  |  |  |
Hair dryer |  |  |  |  |
Hair dryer |  |  |  |  |
Breathing |  |  |  |  |
|  (a) Input events |  (b) GT (Flow from frames) |  (c) MCM [26] (events only) |  (d) Flow from E2VID frames |  (e) Ours
Figure 9: Qualitative comparison between different flow estimation methods.

Flow accuracy is reported in Tab. IV. We evaluate on illuminated sequences for
valid GT flows from frames (please see Sec. 6.3 for the dark sequences).
Consistently for almost all sequences, the proposed method (“Ours (Poisson)”)
provides the best accuracy compared with the baseline methods. Due to the
nature of schlieren, the GT flow magnitude has normally subpixel values.
Hence, we find that the angular error (AE) is a more reliable metric for the
purpose of this benchmark. The largest magnitude of the displacement
($\approx$ 3 px) is observed in the hotplate sequences. Still, it is
remarkable that the proposed method achieves AEE $<1$ pixel. We acknowledge
that the proposed method utilizes both event and frame data, while the
baselines use only event data as input. This is further discussed in Sec. 6.6.

Also, it is noticeable that the Poisson-parameterized estimation (“Ours
(Poisson)”) results in better accuracy than the flow-parameterized estimation
(“Ours (Flow)”). This clearly states the effectiveness of our physically-
motivated parameterization. It provides not only a smaller number of
parameters, as discussed in Sec. 4, but also contributes with better accuracy.

Additionally, we observe that the forced convection usually has a smaller
displacement magnitude than natural convection. This is because the optical
flow $\mathbf{v}$, which we evaluate on, is the temporal derivative of the
density gradient. In the forced convection case (e.g., hair dryer (ON)), the
spatio-temporal changes of the air density at a pixel might be smaller than in
the natural, heat-induced schlieren, since the advection of the flow is
dominant, which can be seen as nearly constant.

Figure 9 shows qualitative results. Although the GT flow is based on a
classical, general-purpose estimation method, it provides remarkably
reasonable flow. The baseline methods (MCM and E2VID) fail to estimate
reasonable flow from events. Especially, we find the alignment-based method
[26] fails to estimate schlieren flow. This is because most events are
generated at the edges of the background pattern, resulting in an uneven
spatial distribution despite air being actually moving, and consequently,
triggering more flickering events. The E2VID-based method surprisingly
reconstructs edge structures of the background pattern (see also Sec. 6.7) in
spite of this specific (flickering) event input, and estimates comparable
flow. However, it fails to recover the fine structure of the flow. Finally,
the flow estimated by our method resembles the GT flow the most, and it even
seems to capture more fine-scale (high-frequency) structures.

###  6.3 HDR Experiment

So far we have established that the proposed method is able to recover the
fine flow structure of the schlieren object. However, schlieren based on
events has another interesting aspect: as shown on the left column of Fig. 9,
the existence of schlieren is already visible in the event data histogram. By
contrast, the schlieren structure is not visible to the naked eye on the raw
frame data but only as the result of optical flow processing. The fact, that
schlieren is observable in a more direct way using events, allows us to
leverage the advantages of the event camera itself, such as HDR and high
temporal resolution.

(a) Frame-based BOS

(b) Ours

Figure 10: _Schlieren imaging under poor illumination (225 $\mathrm{lx}$)_.
(a) Frame-based methods suffer from the limited dynamic range of the frames,
resulting in unrealistic flows with artifacts despite using all grayscale
range available for the frames (normalization). (b) The proposed method
produces a realistic flow, similar to the event data, which is visible due to
the HDR nature of events.

Figure 10 shows qualitative results of the frame-based and event-based
schlieren imaging under poor illumination. The frame-based schlieren method
fails to estimate realistic flow under such conditions, as it needs intense
lighting sources, especially if high-speed cameras are used. Due to the
insufficient brightness, the quality of the frames collapses even after
normalization (i.e., using the entire grayscale range). On the other hand, the
event data captures the schlieren structure (Fig. 10, top right). Furthermore,
the proposed algorithm combining events and frames is surprisingly robust
against such low-quality image inputs. Using natural light (225 $\mathrm{lx}$)
the result (Fig. 10, bottom right) shows the potential of event cameras to
push the limits for future BOS applications. We further discuss the effect of
the amount of illumination in Sec. 6.6.

###  6.4 Super-Slow Motion

Event-based BOS also enables us to see the schlieren at markedly higher
temporal resolution (i.e., slow motion) than conventional frames. To this end,
we conduct a streak-schlieren analysis [3]. The streak analysis focuses on a
single column of the schlieren image to see how it evolves in time, by showing
an $x-t$ diagram (kymogram) of the air convection. The frame-based schlieren
method uses for example Poisson images as schlieren images. For event-based
methods, schlieren images can be either Poisson images or simply event
histograms. Figure 11 shows a comparison of kymograms obtained from frames at
120 Hz (the frame rate) and obtained from events (10$\times$ higher rate,
i.e., at 1200 Hz). Event-based BOS can provide high temporal resolution
kymograms due to the asynchronous nature of event data. Compared with the
frame-based analysis (Fig. 11(a)), the event-based one (Fig. 11(b)) shows
thinner lines of schlieren in space-time. The slow motion schlieren
visualization is best viewed in the supplementary video.

(a) Frame-based kymogram

(b) Event-based kymogram

Figure 11: _Kymograms (space-time plots) and velocimetry_ for a 0.5 second
excerpt of a hotplate sequence. (a) The frame-based schlieren imaging is
limited to the temporal resolution of the camera (120 Hz). (b) The event-based
schlieren can recover higher temporal resolution (e.g., 1200 Hz) thanks to its
data characteristics.

###  6.5 Velocimetry

One can perform velocimetry by fitting curves to the kymograms [3]. Let us
analyse the speed of propagation of schlieren ($\partial{\rho}/\partial{t}$ in
the case of Poisson image) along one direction (e.g., vertical). Figure 11(b)
shows an example on the hot plate sequence. By fitting a curve (line), the
flow propagates 166 px during approximately 68.8 milliseconds. The geometry of
the BOS setup (focal length $f=25$\mathrm{mm}$$, distance to object
$Z_{A}=1.7$\mathrm{m}$$, pixel size $4.86$\mathrm{\SIUnitSymbolMicro m}$$)
leads to an approximate velocity of
$0.805\,219\,3\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$.

###  6.6 Dependency on Frames

4000 $\mathrm{lx}$ |  |  |  |
---|---|---|---|---
2000 $\mathrm{lx}$ |  |  |  |
1000 $\mathrm{lx}$ |  |  |  |
500 $\mathrm{lx}$ |  |  |  |
225 $\mathrm{lx}$ |  |  |  |
110 $\mathrm{lx}$ |  |  |  |
$<$ 50 $\mathrm{lx}$ |  |  |  |
|  (a) Frames |  (b) Flow from frames |  (c) Events |  (d) Ours
Figure 12: _Ablation study for different illumination levels._ Flow (b) uses
normalized frames of the original ones (a) as input, while our method (d) uses
events (c) and the original frames. The frame-based flow deteriorates at
around 500 $\mathrm{lx}$, while events and the estimated flow capture the
schlieren even at illumination levels as low as 225 $\mathrm{lx}$ and 110
$\mathrm{lx}$.

The proposed method uses events and frames. Naturally, the question arises to
which extent the algorithm relies on which signal. To this end, we present the
ablation study with different brightness levels (see also Sec. 6.3). Figure 12
shows the qualitative results for both: frame-based method and our method
(frame plus events), for different illumination levels (measured with a
Voltcraft MS-1300 light meter). As clearly shown, the frame-based flow (column
(b)) starts to deteriorate when the illumination is 1000 $\mathrm{lx}$ or
smaller. For a better performance, we even normalize the range of the frames
used (the exposure time is fixed to maintain the frame rate of 120 fps).
However, this does not provide significantly better results that can compete
with those of our method. By contrast, the following two points are remarkable
about our method: ($i$) schlieren is still visible at 110 $\mathrm{lx}$ in the
event histograms, indicating the HDR capabilities of the noisy input data
(column (c)), and ($ii$) the estimated flow (column (d)) still looks
reasonable when the illumination is as low as 225 $\mathrm{lx}$, despite our
method using the naturally darker frame as an input (column (a)). Note that
our method does not work when the frame is completely black (less than 50
$\mathrm{lx}$). All the above indicates that the proposed method requires
frames, but it can overcome the limited dynamic range of the frames due to the
HDR advantages of event cameras.

###  6.7 Towards a Frame-Free Method

The proposed method utilizes the information from events and a frame, however
the quality of the frame data does not need to be the best, as shown in the
previous section. Hence, an interesting challenge is to replace frame data
with intensity reconstruction from events, such that the proposed method could
be extended to be _frame-free_. To this end, Figure 13 shows the comparison of
the different input frames. Instead of using an acquired frame as an input to
the proposed method, we reconstruct intensity images using E2VID [65] and feed
them as input. Despite the large visual difference between the two different
inputs, the output flow and Poisson images seem to have similar structures.
Although we do not further investigate the quality of the intensity
reconstruction, the results show future possible extension towards frame-free
event-based BOS methods.

(a) Input

(b) Output Poisson

(c) Output Flow

Figure 13: _Towards a frame-free method_. The top row shows the originally
proposed method with the frame-based camera input. The bottom row shows an
E2VID-reconstructed image as the alternative input. In spite of the large
quality difference between the two inputs (a), the output Poisson and flow
images have some visual similarities (b,c).

###  6.8 Effect of the Regularizers and the Translation Field

Ablation. To assess the importance of the regularization and the translation
field parameters $\mathbf{p}$, we conduct an ablation study. The top half of
Tab. V reports optical flow accuracy of the proposed method, the one without
regularization, and the one without the translation. There is a significant
improvement due to the regularizers: without regularizers, the estimated
$\mathbf{q}$ and $\mathbf{p}$ become not smooth anymore, which leads to
irregular flow estimation. The effect of $\mathbf{p}$ is relatively minor, but
still noticeable.

TABLE V: Effect of the regularizers and the translation field.

Hot plate Crushed Ice Dryer AEE $\downarrow$ %Out $\downarrow$ AE $\downarrow$
AEE $\downarrow$ %Out $\downarrow$ AE $\downarrow$ AEE $\downarrow$ %Out
$\downarrow$ AE $\downarrow$ Ours ($\lambda_{1}=0.5$, $\lambda_{2}=0.1$)
$0.486\,88$ $12.214\,58$ $0.420\,67$ $0.325\,60$ $5.176\,59$ $0.301\,11$
$0.395\,36$ $10.173\,93$ $0.336\,96$ w/o regularizers (i.e.,
$\lambda_{1}=\lambda_{2}=0$) $3.370\,93$ $82.038\,58$ $1.111\,48$ $2.499\,37$
$76.325\,11$ $1.017\,46$ $1.233\,17$ $48.625\,64$ $0.756\,27$ w/o translation
model (i.e., $\mathbf{p}=0$) $0.590\,72$ $18.608\,90$ $0.488\,46$ $0.367\,59$
$7.791\,03$ $0.312\,84$ $0.393\,90$ $10.896\,41$ $0.323\,94$
$\lambda_{1}=0.05$, $\lambda_{2}=0.1$ $0.585\,76$ $14.468\,27$ $0.493\,60$
$0.517\,51$ $11.294\,78$ $0.440\,11$ $0.449\,34$ $11.103\,63$ $0.386\,67$
$\lambda_{1}=1.0$, $\lambda_{2}=0.1$ $0.481\,57$ $10.462\,25$ $0.416\,12$
$0.389\,72$ $3.849\,48$ $0.348\,98$ $0.378\,40$ $7.273\,73$ $0.329\,90$
$\lambda_{1}=0.5$, $\lambda_{2}=0.01$ $0.509\,01$ $11.001\,29$ $0.440\,09$
$0.436\,51$ $5.482\,48$ $0.385\,77$ $0.397\,56$ $7.129\,34$ $0.343\,75$
$\lambda_{1}=0.5$, $\lambda_{2}=1.0$ $0.517\,38$ $11.597\,96$ $0.442\,98$
$0.428\,86$ $5.608\,88$ $0.378\,53$ $0.409\,48$ $8.111\,95$ $0.350\,04$

Sensitivity Analysis. We test different weights for each regularizer
$\lambda_{1},\lambda_{2}$ in (12). The weights are set as follows: we fix one
parameter ($\lambda_{1}=0.5$), and vary $\lambda_{2}$ between $0.01$ and
$1.0$; then we fix the other parameter ($\lambda_{2}=0.1$) and vary
$\lambda_{1}$ between $0.05$ and $1.0$. The flow accuracy is reported in the
bottom half of Tab. V. We observe on-par accuracy when $\lambda_{1}=1.0$ with
respect to the base condition (the top row).

###  6.9 Effect of Event Warping

In some of the previous event-based flow estimation methods [48, 49], warping
the events using the estimated optical flow produces sharp intermediate images
that improve convergence (e.g., of the image reconstruction task). We test the
possible efficacy of such an event warping in the BOS setting. As shown in
Fig. 14, warping events with the estimated flow does not have a sharpening
effect in schlieren. This is because: ($i$) event warping leads to sharpening
_if_ events are generated by moving edges (e.g., [26]), which does not hold
true in BOS (see also the large errors of the “MCM” method in Figs. 9 and IV),
and ($ii$) the time window between consecutive frames (e.g., at 120 fps) is
small enough to already produce sharp brightness increment images. Hence, we
do not examine further sharpness/deblur-based approaches.

(a) Original events

(b) Warped events

Figure 14: _Event warping results using the estimated flow._ Event warping
does not improve the sharpness of the data as explained in Sec. 6.9.

##  7 Studies on a Helium Jet Experiment

The theoretical connection between events and schlieren that we establish in
(6) highlights an important difference in the experimental settings between
frame-based and event-based BOS: event-based BOS focuses on the temporal
derivative of the air density, which requires a time-dependent flow. Steady
flows that are often used for frame-based BOS (e.g., supersonic flow) do not
satisfy the time dependency. More precisely, “steady” flow in frame-based BOS
means that it can be averaged over multiple frames (i.e.,
“$\partial{\rho}/\partial{\mathbf{x}}$” in Tab. I) to improve the signal-to-
noise ratio. This difference makes it difficult to apply the experimental
knowledge of common frame-based BOS data to the event-based one. Nonetheless,
we conduct a simulation experiment to make a stronger connection between these
complementary schlieren methods. To this end, we use a frame-based BOS dataset
that is well documented through in experimental evaluation [3]. The dataset
consists of high-speed frames ($6000$ fps) recording a round turbulent helium
jet in air, whose flow has self-similarity (axisymmetric) properties. Each
sequence has 3000 images, of 1024 $\times$ 512 px and with
167$\mathrm{\SIUnitSymbolMicro s}$ exposure time, i.e., 0.5$\mathrm{s}$ of
data.

(a) Events

(b) Similarity analysis

(c) Velocity analysis

Figure 15: _Helium jet experiment_ (image data comes from [3]). (a) Event data
(simulated from frames) and geometry of the scene. (b) Similarity analysis of
jet radial profiles for simulated events. Different colors indicate different
distances from the nozzle, $x/d=\\{59,69,79,89,99\\}$. (c) Velocimetry at the
center of the jet nozzle: the velocity estimated from the simulated events
follows the $1/x$-type decay. Following the same notation as [3, Fig. 7,8],
$x/d$ is the distance normalized by the size of the nozzle (d=1.4mm),
$r/r_{1/2}$ is the radius normalized by the jet half-width, $U_{0}$ is the
estimated velocity profile along the axis of the jet, and $U$ is the estimated
velocity (parallel to the jet axis) at each pixel.

We analyze the velocimetry in accordance with the self-similarity property
using simulated events. The analyzed sequence is the jet with Reynolds number
$Re_{d}=5,980$. First, we run ESIM [66] to simulate events from the high-speed
frames. We use a contrast sensitivity of $0.05$ for both positive and negative
events to have a reasonable signal-to-noise ratio. Events are shown in Fig.
15(a). Then, velocimetry (as in Sec. 6.5) provides the velocity at different
pixel locations on the image plane. Instead of manually fitting lines to the
kymogram using a drawing software as in [3], we detect lines automatically, as
follows: first, we extract patches along the $x/d$ axis of the image plane
(e.g., 100$\times$100 px); then, we smooth the patches with a Gaussian filter
and estimate the slope of the dominant direction within each patch
iteratively, by rotating the patch and finding the angle that maximizes the
magnitude of the rotated patch gradient in a predefined direction.

Figure 15(b) shows the self-similarity by analyzing the simulated events.
Similarly to [3, Fig. 7], the estimated velocity is symmetric and consistent
with the theoretical values (dashed line) along the relative distance using
the jet spreading rate. We observe larger error accumulation than the purely
frame-based method in [3], which can be attributed, among other factors, to
simulation inaccuracies. Figure 15(c) shows the comparison of the velocity at
the center of the jet nozzle along with the distance. Note that the observed
jet centerline velocity is the convective velocity of large scale turbulent
structures (viewed along the line of sight), which is not necessarily the mean
fluid velocity. Nonetheless, the velocity values are similar and reasonably
close to the results in [3, Fig. 8]. Moreover, they seem to follow the
$1/x$-type decay, which agrees with the theoretical model of turbulent flow:
The frame-based results in [3, Fig. 8] degrade approximately for $x/d\leq 80$,
while the simulated event-based results provide a better fit here.

Kymograms are shown in Fig. 16. The curve patterns in the event-based kymogram
look consistent with the frame-based kymogram in [3, Fig. 9], although they
show some artifacts due to the simulation. As mentioned in [3], even $6000$
fps is not enough for the pixels close to the nozzle (e.g., $x/d<20$, where
the flow is very fast. Hence, the event data that is simulated using the
frames inherit such a limitation: the kymogram becomes noisier for smaller
$x/d$ (closer to the nozzle). The high temporal resolution of event cameras
could help overcome such a rate limit of frame-based BOS.

Figure 16: _Kymograms_ of the self-similar turbulent flow from the helium jet
experiment. (a) using frame data and (b) using simulated events. The axes
follow the same notation as [3] for ease of comparison with the figures
therein.

##  8 Discussion

Let us summarize some findings of the first event-based BOS technique.

  * •

The megapixel race has enabled new applications for event cameras in the field
of fluid dynamics.

  * •

Event-based BOS can capture the spatio-temporal derivative of the media
density without any particle seeding.

  * •

As event cameras only sense the temporal changes of the scene, they capture
incremental changes of schlieren, as opposed to changes with respect to a
reference frame.

  * •

Schlieren can be observed directly in the form of flickering events, while it
is more indirect to perceive in frames.

  * •

Event-based BOS can overcome the limitations of dynamic range and temporal
resolution of traditional frame-based BOS.

  * •

Due to the data sparsity, event-based BOS is approximately ten times more data
efficient (smaller storage size). However, estimating the density fluctuations
is more challenging than using frames.

  * •

The knowledge of the background pattern (frames) can be used to handle the
problems of sparsity and noise.

  * •

The proposed method casts the problem into an optical flow estimation one
(i.e., in the realm of computer vision).

  * •

State-of-the-art optical flow methods (e.g., DNN) do not work well for
schlieren because of the very different data properties (e.g., edge motions
vs. complex density fluctuations).

  * •

Directly parameterizing the temporal derivative of the density reduces the
complexity of the problem, provides a link with the physical meaning of the
variables and yields the most accurate results.

  * •

Event-based BOS enables high-temporal resolution velocimetry analysis using
kymograms.

Limitations. The proposed BOS technique using events shows advantages over
frame-based BOS in terms of HDR capabilities and temporal resolution, lowering
the demand for bright illumination and high-speed cameras. However, in other
aspects, it inherits the limitations of frame-based BOS. Optically, the
estimated brightness gradient is a mean value integrated along the optical
axis, and the technique inherently has a trade-off between the observed
displacement and the obtained sharpness of the gradient under investigation.

Additionally BOS is sensitive to vibrations, due to the underlying assumption
that the small perceived changes are only caused by refractive index
variations. Specific to event cameras is that the signal is noisy, and careful
tuning of the camera’s biases is necessary. The proposed method furthermore
relies on a combination of events and frames, thus an accurate spatio-temporal
alignment of both data sources is required. The flow estimation method does
not run in real time. However, raw events visualized as histograms can be
computed online and resemble schlieren images. While the proposed multi-scale
approach improves convergence of the optimization, it limits the spatial
resolution of the flow, which is a similar limitation as in frame-based BOS.

Since event-based BOS relies on temporal changes, flows are observed well if
they are time-dependent. This property makes it difficult to use steady flows
to validate event-based BOS, and hence to evaluate frame- and event-based BOS
in strictly the same settings. Also, event-based BOS cannot improve the
signal-to-noise ratio like the frame-based one does by averaging multiple
frames. Frame-based BOS and event-based BOS have their own strengths, which
may be complementary (e.g., frame-based BOS is good for steady flows, whereas
event-based BOS is good for time-dependent flows). This could be further
investigated in the future.

##  9 Conclusion

We have presented the first event-based BOS imaging technique and an algorithm
to estimate the temporal derivative of the air density gradient. The approach
has been obtained in a mathematically rigorous way and has a physically-
motivated parameterization. Using the frame-based method in well-lit
conditions as ground truth, the experiments evidenced that our approach
outperforms all other tested methods. We furthermore illustrated how the
advantages of event cameras could be leveraged for BOS applications, lowering
the requirements for high illumination and visualizing the turbulent eddies at
a significantly higher temporal resolution. Additional studies on a high-speed
schlieren dataset of a helium jet provided further validation of the method.
We release the code and dataset to the public and hope that this research
opens up new possibilities for the computer vision and fluid dynamics
communities.

## Acknowledgment

We would like to thank Prof. Dr. A. Liberzon at Tel Aviv University for useful
discussions and advice. We also thank the anonymous reviewers for valuable
suggestions. This research was funded by the German Academic Exchange Service
(DAAD), Research Grant-Bi-nationally Supervised Doctoral Degrees/Cotutelle,
2021/22 (57552338) and the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) under Germany’s Excellence Strategy – EXC 2002/1 “Science
of Intelligence” – project number 390523135.

## References

  * [1] G. S. Settles, _Schlieren and shadowgraph techniques: visualizing phenomena in transparent media_. Springer Science & Business Media, 2001.
  * [2] G. S. Settles and M. J. Hargather, “A review of recent developments in schlieren and shadowgraph techniques,” _Meas. Sci. Technol._ , vol. 28, no. 4, p. 042001, 2017.
  * [3] G. S. Settles and A. Liberzon, “Schlieren and BOS velocimetry of a round turbulent helium jet in air,” _Optics and Lasers in Eng._ , vol. 156, p. 107104, 2022.
  * [4] P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128$\times$128 120 dB 15 $\mu$s latency asynchronous temporal contrast vision sensor,” _IEEE J. Solid-State Circuits_ , vol. 43, no. 2, pp. 566–576, 2008.
  * [5] T. Finateu, A. Niwa, D. Matolin, K. Tsuchimoto, A. Mascheroni, E. Reynaud, P. Mostafalu, F. Brady, L. Chotard, F. LeGoff, H. Takahashi, H. Wakabayashi, Y. Oike, and C. Posch, “A 1280x720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86$\mu$m pixels, 1.066Geps readout, programmable event-rate controller and compressive data-formatting pipeline,” in _IEEE Int. Solid-State Circuits Conf. (ISSCC)_ , 2020, pp. 112–114.
  * [6] G. Gallego, T. Delbruck, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. Davison, J. Conradt, K. Daniilidis, and D. Scaramuzza, “Event-based vision: A survey,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 44, no. 1, pp. 154–180, 2022.
  * [7] M. Raffel, C. E. Willert, J. Kompenhans _et al._ , _Particle image velocimetry: a practical guide_. Springer, 1998, vol. 2.
  * [8] Y. Ding, Z. Li, Z. Chen, Y. Ji, J. Yu, and J. Ye, “Full-volume 3d fluid flow reconstruction with light field piv,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2023.
  * [9] H. Richard and M. Raffel, “Principle and applications of the background oriented schlieren (BOS) method,” _Meas. Sci. Technol._ , vol. 12, no. 9, p. 1576, 2001.
  * [10] E. Goldhahn and J. Seume, “The background oriented schlieren technique: sensitivity, accuracy, resolution and application to a three-dimensional density field,” _Experiments in fluids_ , vol. 43, pp. 241–249, 2007.
  * [11] B. Atcheson, W. Heidrich, and I. Ihrke, “An evaluation of optical flow algorithms for background oriented schlieren imaging,” _Exp. Fluids_ , vol. 46, pp. 467–476, 2009.
  * [12] B. E. Schmidt and M. R. Woike, “Wavelet-based optical flow analysis for background-oriented schlieren image processing,” _AIAA Journal_ , vol. 59, no. 8, pp. 3209–3216, 2021.
  * [13] R. Hooke, “Of a new property in the air,” _Micrographia, Observation LVIII_ , pp. 217–219, 1665.
  * [14] M. Raffel, “Background-oriented schlieren (BOS) techniques,” _Exp. Fluids_ , vol. 56, no. 3, pp. 1–17, 2015.
  * [15] A. D. Gardner, M. Raffel, C. Schwarz, J. N. Braukmann, and C. C. Wolf, “Reference-free digital shadowgraphy using a moving bos background,” _Exp. in Fluids_ , vol. 61, pp. 1–5, 2020.
  * [16] P. Krehl and S. Engemann, “August toepler—the first who visualized shock waves,” _Shock Waves_ , vol. 5, pp. 1–18, 1995.
  * [17] O. Sommersel, D. Bjerketvedt, S. Christensen, O. Krest, and K. Vaagsaether, “Application of background oriented schlieren for quantitative measurements of shock waves from explosions,” _Shock Waves_ , vol. 18, pp. 291–297, 2008\.
  * [18] K. Hayasaka, Y. Tagawa, T. Liu, and M. Kameda, “Optical-flow-based background-oriented schlieren technique for measuring a laser-induced underwater shock wave,” _Experiments in Fluids_ , vol. 57, pp. 1–11, 2016\.
  * [19] M. J. Hargather and G. S. Settles, “Natural-background-oriented schlieren imaging,” _Experiments in fluids_ , vol. 48, no. 1, pp. 59–68, 2010.
  * [20] J. T. Heineck, D. W. Banks, N. T. Smith, E. T. Schairer, P. S. Bean, and T. Robillos, “Background-oriented schlieren imaging of supersonic aircraft in flight,” _AIAA Journal_ , vol. 59, no. 1, pp. 11–21, 2021.
  * [21] J. H. Tu, C. W. Rowley, D. M. Luchtenburg, S. L. Brunton, and J. N. Kutz, “On dynamic mode decomposition: Theory and applications,” _J. Computational Dynamics_ , vol. 1, no. 2, pp. 391–421, 2014.
  * [22] L. Venkatakrishnan and G. Meier, “Density measurements using the background oriented schlieren technique,” _Experiments in Fluids_ , vol. 37, pp. 237–247, 2004.
  * [23] R. Benosman, C. Clercq, X. Lagorce, S.-H. Ieng, and C. Bartolozzi, “Event-based visual flow,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 25, no. 2, pp. 407–417, 2014.
  * [24] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “EV-FlowNet: Self-supervised optical flow estimation for event-based cameras,” in _Robotics: Science and Systems (RSS)_ , 2018, pp. 1–9.
  * [25] J. J. Hagenaars, F. Paredes-Valles, and G. C. H. E. de Croon, “Self-supervised learning of event-based optical flow with spiking neural networks,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 34, 2021, pp. 7167–7179.
  * [26] S. Shiba, Y. Aoki, and G. Gallego, “Secrets of event-based optical flow,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2022, pp. 628–645.
  * [27] ——, “Fast event-based optical flow estimation by triplet matching,” _IEEE Signal Process. Lett._ , pp. 1–5, 2023.
  * [28] G. Gallego, H. Rebecq, and D. Scaramuzza, “A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2018, pp. 3867–3876.
  * [29] U. M. Nunes and Y. Demiris, “Robust event-based vision model estimation by dispersion minimisation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2021\.
  * [30] X. Peng, L. Gao, Y. Wang, and L. Kneip, “Globally-optimal contrast maximisation for event cameras,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 44, no. 7, pp. 3479–3495, 2022.
  * [31] S. Shiba, Y. Aoki, and G. Gallego, “A fast geometric regularizer to mitigate event collapse in the contrast maximization framework,” _Adv. Intell. Syst._ , p. 2200251, 2022.
  * [32] H. Kim, S. Leutenegger, and A. J. Davison, “Real-time 3D reconstruction and 6-DoF tracking with an event camera,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2016, pp. 349–364.
  * [33] H. Rebecq, T. Horstschäfer, G. Gallego, and D. Scaramuzza, “EVO: A geometric approach to event-based 6-DOF parallel tracking and mapping in real-time,” _IEEE Robot. Autom. Lett._ , vol. 2, no. 2, pp. 593–600, 2017\.
  * [34] A. Z. Zhu, N. Atanasov, and K. Daniilidis, “Event-based visual inertial odometry,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2017, pp. 5816–5824.
  * [35] J. Hidalgo-Carrió, G. Gallego, and D. Scaramuzza, “Event-aided direct sparse odometry,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , Jun. 2022, pp. 5781–5790.
  * [36] X. Zhang and L. Yu, “Unifying motion deblurring and frame interpolation with events,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , Jun. 2022, pp. 17 765–17 774.
  * [37] S. Tulyakov, A. Bochicchio, D. Gehrig, S. Georgoulis, Y. Li, and D. Scaramuzza, “Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , Jun. 2022, pp. 17 755–17 764.
  * [38] Y. Gao, S. Li, Y. Li, Y. Guo, and Q. Dai, “Superfast: 200$\boldsymbol{\times}$ video frame interpolation via event camera,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 45, pp. 7764–7780, Jun. 2023.
  * [39] Y. Suh, S. Choi, M. Ito, J. Kim, Y. Lee, J. Seo, H. Jung, D.-H. Yeo, S. Namgung, J. Bong, J. seok Kim, P. K. J. Park, J. Kim, H. Ryu, and Y. Park, “A 1280x960 Dynamic Vision Sensor with a 4.95-$\mu$m pixel pitch and motion artifact minimization,” in _IEEE Int. Symp. Circuits Syst. (ISCAS)_ , 2020, pp. 1–5.
  * [40] B. Pfrommer, “Frequency Cam: Imaging periodic signals in real-time,” in _arXiv e-prints_ , 2022.
  * [41] C. E. Willert and J. Klinner, “Event-based imaging velocimetry: an assessment of event-based cameras for the measurement of fluid flows,” _Exp. Fluids_ , vol. 63, no. 6, pp. 1–20, 2022.
  * [42] Y. Wang, R. Idoughi, and W. Heidrich, “Stereo event-based particle tracking velocimetry for 3D fluid flow reconstruction,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2020, pp. 36–53.
  * [43] https://github.com/uzh-rpg/event-based_vision_resources, 2017.
  * [44] D. Gehrig, H. Rebecq, G. Gallego, and D. Scaramuzza, “EKLT: Asynchronous photometric feature tracking using events and frames,” _Int. J. Comput. Vis._ , vol. 128, pp. 601–618, 2020.
  * [45] G. Gallego, C. Forster, E. Mueggler, and D. Scaramuzza, “Event-based camera pose tracking using a generative event model,” 2015, arXiv:1510.01972.
  * [46] S. Bryner, G. Gallego, H. Rebecq, and D. Scaramuzza, “Event-based, direct camera tracking from a photometric 3D map using nonlinear optimization,” in _IEEE Int. Conf. Robot. Autom. (ICRA)_ , 2019.
  * [47] L. Pan, M. Liu, and R. Hartley, “Single image optical flow estimation with an event camera,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2020, pp. 1669–1678.
  * [48] F. Paredes-Valles and G. C. H. E. de Croon, “Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy,” in _IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)_ , 2021, pp. 3445–3454.
  * [49] Z. Zhang, A. Yezzi, and G. Gallego, “Formulating event-based image reconstruction as a linear inverse problem with deep regularization using optical flow,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2022.
  * [50] R. Graca and T. Delbruck, “Unraveling the paradox of intensity-dependent DVS pixel noise,” in _Int. Image Sensor Workshop (IISW)_ , 2021.
  * [51] C. Scheerlinck, N. Barnes, and R. Mahony, “Continuous-time intensity estimation using event cameras,” in _Asian Conf. Comput. Vis. (ACCV)_ , 2018, pp. 308–324.
  * [52] C. Brandli, L. Muller, and T. Delbruck, “Real-time, high-speed video decompression using a frame- and event-based DAVIS sensor,” in _IEEE Int. Symp. Circuits Syst. (ISCAS)_ , 2014, pp. 686–689.
  * [53] A. Z. Zhu, D. Thakur, T. Ozaslan, B. Pfrommer, V. Kumar, and K. Daniilidis, “The multivehicle stereo event camera dataset: An event camera dataset for 3D perception,” _IEEE Robot. Autom. Lett._ , vol. 3, no. 3, pp. 2032–2039, Jul. 2018.
  * [54] M. Gehrig, W. Aarents, D. Gehrig, and D. Scaramuzza, “DSEC: A stereo event camera dataset for driving scenarios,” _IEEE Robot. Autom. Lett._ , vol. 6, no. 3, pp. 4947–4954, 2021.
  * [55] S. N. Barkas, “An introduction to fast Poisson solvers,” _Philips J Res_ , vol. 37, no. 5-6, pp. 231–264, 2005.
  * [56] H. Kim, A. Handa, R. Benosman, S.-H. Ieng, and A. J. Davison, “Simultaneous mosaicing and tracking with an event camera,” in _British Mach. Vis. Conf. (BMVC)_ , 2014.
  * [57] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” _Int. Conf. Learn. Representations (ICLR)_ , 2015.
  * [58] C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck, “A 240x180 130dB 3$\mu$s latency global shutter spatiotemporal vision sensor,” _IEEE J. Solid-State Circuits_ , vol. 49, no. 10, pp. 2333–2341, 2014.
  * [59] G. Taverni, D. P. Moeys, C. Li, C. Cavaco, V. Motsnyi, D. S. S. Bello, and T. Delbruck, “Front and back illuminated Dynamic and Active Pixel Vision Sensors comparison,” _IEEE Trans. Circuits Syst. II (TCSII)_ , vol. 65, no. 5, pp. 677–681, 2018.
  * [60] M. Muglikar, M. Gehrig, D. Gehrig, and D. Scaramuzza, “How to calibrate your event camera,” in _IEEE Conf. Comput. Vis. Pattern Recog. Workshops (CVPRW)_ , 2021, pp. 1403–1409.
  * [61] F. Hamann and G. Gallego, “Stereo co-capture system for recording and tracking fish with frame- and event cameras,” in _26th Int. Conf. on Pattern Recognition (ICPR), Visual observation and analysis of Vertebrate And Insect Behavior (VAIB) Workshop_ , 2022.
  * [62] G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in _Scandinavian Conf. on Im. Analysis (SCIA)_ , 2003, pp. 363–370.
  * [63] Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai, and H. Li, “FlowFormer: A transformer architecture for optical flow,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2022, pp. 668–685.
  * [64] Z. Huang, T. Zhang, W. Heng, B. Shi, and S. Zhou, “Real-time intermediate flow estimation for video frame interpolation,” in _Eur. Conf. Comput. Vis. (ECCV)_ , 2022, pp. 624–642.
  * [65] H. Rebecq, R. Ranftl, V. Koltun, and D. Scaramuzza, “High speed and high dynamic range video with an event camera,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 43, no. 6, pp. 1964–1980, 2021.
  * [66] H. Rebecq, D. Gehrig, and D. Scaramuzza, “ESIM: an open event camera simulator,” in _Conf. on Robotics Learning (CoRL)_ , ser. Proc. Machine Learning Research, vol. 87. PMLR, 2018, pp. 969–982.

|  Shintaro Shiba received the Ph.D. degree in Engineering from Keio
University, Japan, in 2023. From 2021 to 2023, he was a visiting researcher at
the Robotics Interactive Perception Laboratory, Technische Universität Berlin,
supported by a German Academic Exchange Service (DAAD) bi-national doctoral
research fellowship. He received his Master’s in Cognitive Neuroscience from
the University of Tokyo in 2017. His research interests include computer
vision, machine learning, robotics, and neuroscience.
---|---
|  Friedhelm Hamann is a Ph.D. student at the Robotic Interactive Perception
Laboratory, Technical University Berlin, and a member of the Science of
Intelligence Excellence Cluster, Berlin, Germany. He received a master’s
degree in Electrical Engineering, Computer Engineering, and Information
Technology from RWTH Aachen University in 2021. His research interests include
computer vision, machine learning, and signal processing.
---|---
|  Yoshimitsu Aoki received the Ph.D. degree in engineering from Waseda
University in 2001. From 2002 to 2008, he was an Associate Professor with the
Department of Information Engineering, Shibaura Institute of Technology. He is
currently a Professor with the Department of Electronics and Electrical
Engineering, Keio University. He performs research in the areas of computer
vision, pattern recognition, and media understanding.
---|---
|  Guillermo Gallego (SM’19) is Associate Professor at Technische Universität
Berlin, in the Dept. of Electrical Engineering and Computer Science, and at
the Einstein Center Digital Future, Berlin, where he leads the Robotic
Interactive Perception Laboratory. He is also a Principal Investigator at the
Science of Intelligence Excellence Cluster, Berlin, Germany. He received the
PhD degree in Electrical and Computer Engineering from the Georgia Institute
of Technology, USA, in 2011, supported by a Fulbright Scholarship. From 2011
to 2014 he was a Marie Curie researcher with Universidad Politecnica de
Madrid, Spain, and from 2014 to 2019 he was a postdoctoral researcher at the
Robotics and Perception Group, University of Zurich and ETH Zurich,
Switzerland. Since 2022 he is also Co-Director of the HEIBRiDS
interdisciplinary research school, Berlin. His research interests include
robotics, computer vision, signal processing, optimization and geometry.
---|---