# Lightweight Event-based Optical Flow Estimation via Iterative Deblurring

Yilun Wu Federico Paredes-Vallés Guido C. H. E. de Croon
Micro Air Vehicle Laboratory, Technische Universiteit Delft
Delft, The Netherlands
{Y.Wu-9, F.ParedesValles<EMAIL_ADDRESS>
###### Abstract

Inspired by frame-based methods, state-of-the-art event-based optical flow
networks rely on the explicit computation of correlation volumes, which are
expensive to compute and store on systems with limited processing budget and
memory. To this end, we introduce IDNet (Iterative Deblurring Network), a
lightweight yet well-performing event-based optical flow network without using
correlation volumes. IDNet leverages the unique spatiotemporally continuous
nature of event streams to propose an alternative way of implicitly capturing
correlation through iterative refinement and motion deblurring. Our network
does not compute correlation volumes but rather utilizes a recurrent network
to maximize the spatiotemporal correlation of events iteratively. We further
propose two iterative update schemes: “ID” which iterates over the same batch
of events, and “TID” which iterates over time with streaming events in an
online fashion. Benchmark results show the former “ID” scheme can reach close
to state-of-the-art performance with 33% of savings in compute and 90% in
memory footprint, while the latter “TID” scheme is even more efficient
promising 83% of compute savings and 15 times less latency at the cost of 18%
of performance drop.

##  1 Introduction

Figure 1: Illustration of the IDNet pipeline for temporal iterative deblurring
(i.e. TID scheme). Raw events are first deblurred according to the initial
coarse optical flow estimate $\hat{\mathcal{F}}^{t}$ before being processed by
the backbone RNN. The RNN extracts the residual motion from the deblurred
events and outputs the residual flow $\Delta\mathcal{F}^{t}$ which is added to
the initial estimate $\hat{\mathcal{F}}^{t}$ to arrive at the final flow
estimation $\mathcal{F}^{t}$. The RNN additionally proposes a coarse estimate
$\hat{\mathcal{F}}^{t+1}$ for the next timestep under continuous operation.

Optical flow estimation, i.e. estimating the apparent motion vector of pixels
through time on the image plane, is both a central and challenging task in
computer vision. As it encodes a primitive form of motion information, optical
flow serves as the backbone for many applications ranging from video
compression [24, 39] and computational photography [20, 38] to robotic
navigation [31, 5]. Traditionally, optical flow is estimated from frame-based
cameras with methods ranging from hand-crafted optimization [26, 14] to modern
learning-based methods [7, 36, 37]. Compared to frame-based cameras, event
cameras capture asynchronous brightness changes in continuous time, offering
high dynamic range measurements free from motion blur even at high speeds and
low lighting conditions [8]. In addition, such data format is superior for
estimating optical flow, as the trajectory of motion is captured continuously
in the data stream instead of being captured in discrete pixel locations
across two frames. To make the raw event stream compatible with network
architectures designed for images, most works first process events into a
dense representation before feeding it into the network [43, 40].

The advances in the recent performance of learning-based methods for optical
flow estimation can be seen as the process of incorporating ever more fitting
inductive biases to the underlying problem. PWC-Net [36] introduces pyramid
processing, warping, and cost volume, three common ingredients used in
classical model-based methods, and achieved superior performance with a much
smaller network compared to its predecessor [7]. RAFT [37] further improves
the performance by replacing the optimization process in finding maximum
correlation with a learning-based RNN operating over the cost volume.

Recently, an event-based version of RAFT, termed E-RAFT [12], has been
developed. The method applied RAFT architecture to event data by treating two
event representations consecutive in time as two images in the context of
estimating flow from frames. E-RAFT achieves state-of-the-art performance on
the challenging event-based optical flow benchmark DSEC [11], demonstrating
the strong effectiveness of appropriate inductive biases (i.e. correlation and
iterative optimization) for this problem.

However, we note that there are two major drawbacks of the method when applied
to event data: i) the multi-bin event representation can only be processed all
at once instead of being processed one bin at a time which allows for
parallelizing the computation as events arrive. This prevents the algorithm
from real-time operations under tight latency constraints; ii) computing and
storing the all-pair correlation volume is costly and scales badly with the
input resolution, hindering the algorithm’s execution on platforms with
limited memory during inference.

We propose IDNet (Iterative Deblurring Network), a lightweight event-based
optical flow algorithm. At its core, IDNet processes event bins sequentially
through a backbone RNN. On top of that, an iterative update scheme is applied
so that flow is continually refined through the same backbone network without
introducing any separate network modules between iterations.

Instead of explicitly computing the correlation volume of dense event
representations, we implicitly model correlation by clustering events which
originate from the same moving object. This clustering process is achieved
through motion compensation (a.k.a. deblurring), as introduced in [9].
Illustrated in Fig. 1, with an appropriate flow estimate, motion compensation
traces the trajectory of pixels by transporting the spatial coordinates of
events through time to a common reference. Compared to raw events, the
deblurred events exhibit higher spatiotemporal correlation between the event
bins, as evinced by the decreased amount of blur shown in the collapsed view
in Fig. 1.

In addition, the operation of motion compensation could be performed in an
iterative fashion. As shown in Fig. 1, at the end of a previous iteration, a
flow estimate is proposed. This prior flow is used to perform an initial
deblur of the raw events. The residual flow within the deblurred events is
then extracted. The prior flow can be refined by incorporating this residual
flow from the current iteration. Other than the iterative update scheme
illustrated in Fig. 1, we also propose another update scheme iterating over
the same batch of events which yields better performance with more compute.
For a more detailed illustration of the pipeline, please refer to Fig. 2.

We compare our methods against previous works on the challenging publicly
available benchmark DSEC [11] and demonstrate competitive performance against
the state-of-the-art method while using a fraction of the compute and memory
footprint during inference. We also show the effectiveness of the two
inductive biases introduced, i.e. iterative processing and deblurring, through
controlled ablation studies.

In summary, our contributions are:

  * •

We introduce the first learning method which combines iterative processing and
motion compensation (deblur) for event-based optical flow estimation.

  * •

We propose two effective modes of performing iterative deblurring. One mode
iteratively deblurs and updates the optical flow for the same batch of events
while the other mode runs continuously and deblurs the events over time.

  * •

The results on public benchmarks along with ablation studies reveal the
competitive performance against the state-of-the-art method, as well as the
superior efficiency in terms of compute and memory savings.

##  2 Related Works

###  2.1 Event-based Optical Flow

Traditionally, optical flow estimation is performed from image frames with
algorithms ranging from classical optimization techniques such as [14, 26] to
modern learning-based methods [7, 32, 36, 17, 37] which combine the powerful
expressivity of deep neural networks with inductive biases such as correlation
matching and coarse-to-fine estimation. While frame-based optical flow has
reached a mature stage after decades of research, there are scenarios where
images would suffer from quality degradation, such as motion blur and
insufficient exposure.

Event cameras register brightness changes with high temporal resolution free
from motion blur even at challenging illumination conditions [8, 4].
Furthermore, the trajectory of motion is captured continuously through space
and time in the event stream rather than being discrete over two frames. This
property makes event streams the superior data type for estimating optical
flow.

The early event-based optical flow method [3] adapts frame-based methods such
as KLT [26] to the event-based domain or utilizes hand-crafted heuristics to
fit event data [2, 27]. [1] jointly optimizes image brightness and optical
flow by exploiting brightness constancy, flow smoothness and the event
generation model together. [9, 34] introduces a framework to recover optical
flow from events by maximizing the contrast of motion-compensated events.
Other than directly optimizing the objective, this principle is also applied
as a learning signal in self-supervised learning methods such as [41, 13, 28,
40].

The availability of event simulators [16, 10] and large-scale datasets [42,
11] enable learning-based methods to achieve superior performance over model-
based ones. [41] introduces an event representation which split events
proportionally to the nearby temporal bins and a U-Net architecture for
processing the event representation. [40] jointly estimates optical flow,
depth and egomotion. Both methods construct multi-level feature pyramids and
[40] warps the event representation slices to evaluate the quality of flow
through the contrast maximization framework. E-RAFT [12], an event-based
version of RAFT [37] is the first learning-based method to introduce
correlation volumes into the event domain. The method computes the 4D all-pair
correlation volume between two neighboring event representations in time,
which is then iteratively processed to arrive at the final optical flow
estimate.

Although E-RAFT achieves state-of-the-art performance while being relatively
lightweight compared to previous learning-based methods, the explicit
computation and storage of all-pair correlation volume can be costly for
resource-constrained compute platforms. In addition, the time and storage
complexity scales badly with input resolution. For two event representations
of size $\text{C}\times\text{H}\times\text{W}$, the compute and storage
complexity of such correlation volume is
$\mathcal{O}({(\text{H}\times\text{W})}^{2}\times C)$ and
$\mathcal{O}({(\text{H}\times\text{W})}^{2})$ respectively. This storage
requirement eliminates the possibility of its deployment on memory-limited
compute platforms and prohibits the algorithm from scaling up to larger input
resolutions.

###  2.2 Iterative Refinement for Optical Flow

The idea of iterative refinement for optical flow can be traced back to the
early works of iterative KLT tracking [26] where Taylor series expansion is
applied to linearize the problem and iteratively solve for the residual error.
Learning-based methods incorporate this principle as well. Most works [32, 36,
17] choose to perform coarse-to-fine refinement along the feature pyramid. In
addition to spatial refinement, other works such as [19, 18, 25, 37] stacks
multiple modules in series to iteratively estimate the residual flow from
representations warped by the current flow estimate. While [19] warps images,
[18] and [37] samples correlation volumes and [25] recomputes correlation
volumes. [25, 18, 37] share weights of the refinement modules, making their
architecture recurrent rather than feed-forward.

Recently, iterative refinement for optical flow is also applied to event-based
data. [23] and [15] uses pyramid scale to perform coarse-to-fine estimation
similar to PWC-Net [36] but do not perform flow refinement through iterations.
[6] adopts the iterative update scheme in [18] and processes the correlation
volume between warped event slices through time sequentially to arrive at a
residual flow estimate. [12] instead directly processes the all-pair
correlation volume through an RNN for multiple iterations and extracts the
final flow estimate from the state of the RNN at the end of its iterations.

All aforementioned methods rely on the explicit construction of correlation
volumes, in part to adapt existing successful architectures in the frame-based
domain to event data. We exploit the unique spatiotemporal continuous nature
of event data by combining iterative refinement and motion compensation. We
later show that such exploitation leads to lightweight solutions which are
also competitive in performance.

###  2.3 Continuous Operation

Since optical flow is temporally highly correlated as the motion in a real-
world environment is mostly smooth and continuous, it makes sense to leverage
the prior estimate from the past to better predict the present optical flow.
However, to the best of our knowledge, only RAFT [37] introduced a “warm-
starting” strategy when applied to video data among popular frame-based
methods. RAFT [37], as well as, E-RAFT [12] directly transports the previous
flow field to the current timestep by the flow itself under the assumption of
constant linear motion.

The prior from past flow estimate could also be leveraged through the memory
of a continuously running recurrent network. Several works [13, 22, 30] have
proposed recurrent architectures to predict event-based optical flow at the
same high update rate of their RNNs. This prevents the network to accumulate
flow over longer timespans and hence would compare unfavorably on public
benchmarks such as DSEC [11] whose ground truth rate is only at 10Hz.

##  3 Methodology

Figure 2: Overall pipeline of IDNet with iterative deblurring scheme (i.e. ID
scheme). At the beginning of iterations, a zero flow estimate is initiated. At
each iteration, the previous flow estimate is used to deblur events. The
deblurred event bins are fed into the RNN sequentially one bin at a time. A
residual flow is estimated and used for deblurring in the next iteration. The
final flow is the sum of all predicted residual flows throughout iterations.
Finally, an L1 loss is enforced between the final flow estimate and ground
truth.

###  3.1 Problem Formulation

We define the problem of optical flow estimation as the mapping from a
sequence of event streams containing triggering timestamp $t$, pixel location
$x,y$ and binary polarity $p$ in a time window:
$\\{e_{i}=(t_{i},x_{i},y_{i},p_{i})\\}$, where
$t_{i}\in[t_{\text{begin}},t_{\text{end}}]$ to a dense optical flow vector for
every pixel: $(u,v)$, which is the two-dimensional flow defined at time
$t=t_{\text{begin}}$. As in [12, 43], we utilize the discretized event volume
as the event representation, which interpolates each event into its two
adjacent bins weighted by its timestamp and polarity.

Specifically, we create a representation
$\mathcal{E}\in\textrm{R}^{\textrm{B}\times\textrm{H}\times\textrm{W}}$ with B
bins out of the event stream $\\{e_{i}=(t_{i},x_{i},y_{i},p_{i})\\}$ of
interest as follows:

$\displaystyle t_{i}^{\ast}$
$\displaystyle=(B-1)(t_{i}-t_{\text{begin}})/(t_{\text{end}}-t_{\text{begin}})$
(1) $\displaystyle\mathcal{E}(B,x,y)$
$\displaystyle=\sum_{i|x_{i}=x,y_{i}=y}p_{i}\max(0,1-|B-t_{i}^{\ast}|)$ (2)

where $\textrm{H},\textrm{W}$ corresponds to the height and width resolution
of the event stream.

###  3.2 Motion Compensation

As introduced in [9], we utilize the principle of motion compensation as a
core step in our processing. Specifically, we use the flow estimate
$\mathcal{F}$ to deblur the events to a reference time $t_{\text{ref}}$ by
changing its pixel coordinate $\mathbf{x}_{i}=(x_{i},y_{i})$ in the following
way:

$\mathbf{x^{\prime}}_{i}=\mathbf{x}_{i}+(t_{\text{ref}}-t_{i})\mathcal{F}(x_{i})$
(3)

Ideally, under the assumption of brightness constancy and linear motion, the
ideal flow would cancel out all the motion in the events, leading to a static
event representation without any motion. An illustration of the comparison
before and after such motion compensation can be seen in Fig. 1. The collapsed
view where all bins of the event representation $\mathcal{E}$ are projected
onto the same 2D plane reveals the deblurred events exhibit less motion and
moving objects appear static and preserve sharp edges.

###  3.3 Iterative Deblurring

The main architecture of IDNet under iterative deblurring (ID scheme) is shown
in Fig. 2. The backbone of our proposed IDNet is a recurrent neural network
which processes the input event bins sequentially one bin at a time. An
optical flow field is read out from the state of the RNN once all bins have
been passed through it. In this case, we select a ConvGRU as the choice for
the recurrent unit.

Next, we introduce iterative deblurring on top of the sequential processing
described above. The design here is inspired by the predictive coding scheme
in the visual cortex [33], where the belief of the world is iteratively
updated and only the error signal, i.e. the difference between the visual
input signal and the predicted signal according to the current belief, is sent
to the visual cortex for refining belief until equilibrium. We take a similar
approach by using motion compensation as the prediction model to iteratively
refine the flow from the residual signal.

input : Event bins $\mathcal{E}$ containing events in a time window of
duration T

Number of deblurring iterations $N$

output : Optical Flow Prediction $\mathcal{F}$ for the motion during the time
window of events

$\mathcal{F}_{0}\leftarrow\mathbf{0}$,
$\mathcal{M}_{\text{RNN},0}\leftarrow\mathbf{0}$,
$\mathcal{E}_{\text{deblur},0}\leftarrow\mathcal{E}$

for  _$i\leftarrow 0;i <N;i$++_ do

$\Delta\mathcal{F}_{i}\leftarrow$
RNN(_Encoder(_$\mathcal{E}_{\text{deblur},i}$_);
$\mathcal{M}_{\text{RNN},i}$_)

$\mathcal{F}_{i+1}\leftarrow\mathcal{F}_{i}+\Delta\mathcal{F}_{i}$

$\mathcal{M}_{\text{RNN},i+1}\leftarrow$
WarmStartModule(_$\mathcal{F}_{i+1}$_)

$\mathcal{E}_{\text{deblur},i+1}\leftarrow$
Deblur(_$\mathcal{E}_{\text{deblur},i};\Delta\mathcal{F}_{i}$_)

$\mathcal{F}\leftarrow\mathcal{F}_{N}$

return _$\mathcal{F}$_

Algorithm 1 Iterative deblurring (ID).

An overview of the algorithm is presented in Algorithm 1. Specifically, as
shown in Fig. 2, we initialize our flow estimate as well as the memory of the
backbone RNN to be zero at the beginning ($\mathcal{F}_{0}=\mathbf{0}$,
$\mathcal{M}_{\text{RNN},0}=0$). Once the backbone RNN makes a prediction
$\mathcal{F}_{1}$ of the flow from the raw event bins $\mathcal{E}_{0}$, the
event bins are motion-compensated with the optical flow vector
$\mathcal{F}_{1}(x_{i})$, resulting in partially deblurred event bins
$\mathcal{E}_{1}$. We then start the next iteration of refinement by passing
$\mathcal{E}_{1}$ into the backbone network for another round of flow
prediction.

To harness the recurrency of the RNN, we initialize the memory of our backbone
RNN through a warm-starting module (WSM) which consists of feed-forward
convolutional layers:
$\mathcal{M}_{\text{RNN},1}=\text{WSM}(\mathcal{F}_{1})$. This time, we update
the previous belief by adding the residual flow $\Delta\mathcal{F}_{2}$
estimated from $\mathcal{E}_{1}$ to the previous belief $\mathcal{F}_{1}$ to
arrive at the flow estimate at iteration 2:
$\mathcal{F}_{2}=\mathcal{F}_{1}+\Delta\mathcal{F}_{2}$, from which a new
iteration could be started again.

The design choice of using the same RNN (weight sharing) for all the
iterations is motivated by the fact that the backbone network is expected to
extract any residual motion in the presented event representation, raw or
deblurred. Hence, the task (input to output mapping) remains the same for all
iterations.

###  3.4 Iterative Deblurring through Time

We also explore another mode of iterative refinement through the temporal
scale. The update schematic is shown in Fig. 1. Instead of deblurring the same
events multiple iterations to predict their corresponding flow, we give the
network access to a continuous stream of events where at each timestep we
deblur the event bins $\mathcal{E}^{t}$ and pass the deblurred bins
$\mathcal{E}_{\text{deblur}}^{t}$ through the backbone RNN only once. In
addition to estimating the optical flow $\mathcal{F}^{t}$ for the current
timestep, we predict an initial flow $\hat{\mathcal{F}}^{t+1}$ for deblurring
the event bins $\mathcal{E}^{t+1}$ from the next timestep. This recurrent
connection allows the network to refine the flow and compose a finer prior
through time. This is evidenced by examples shown in Fig. 3. The key
difference between running this pipeline vs simply passing the event bins
through the backbone network once, is that: firstly, an informative memory of
the backbone RNN is initialized from previous estimates rather than starting
from zero, and secondly, the backbone network only processes deblurred event
bins which exhibit a much smaller motion range compared to raw data. This
helps the backbone RNN to use its limited parameters to focus on improving the
flow accuracy at a finer scale.

Operational-wise, we utilize the same principle of motion compensating events
and warm starting the memory of RNN as before, but introduce an additional
module to estimate the initial flow from the following timestep through the
state of the backbone RNN. A pseudo code of the algorithm is laid out in
Algorithm 2.

input : Event bins $\mathcal{E}^{t}$ containing events with
$\tau\in[t,t+\text{T}]$

Initial flow estimate at $\hat{\mathcal{F}}^{t}$ at time $t$

output : Optical Flow Prediction $\mathcal{F}^{t}$ at time $t$ for the motion
from $\tau\in[t,t+\text{T}]$

$\mathcal{M}_{\text{RNN}}^{t}\leftarrow$
WarmStartModule(_$\hat{\mathcal{F}}^{t}$_)

$\mathcal{E}_{\text{deblur}}^{t}\leftarrow$
Deblur(_$\mathcal{E}^{t};\hat{\mathcal{F}}^{t}$_)

$\Delta\mathcal{F}^{t},\hat{\mathcal{F}}^{t+1}\leftarrow$
RNN(_Encoder(_$\mathcal{E}_{\text{deblur}}^{t}$_) ;
$\mathcal{M}_{\text{RNN}}^{t}$_)

$\mathcal{F}^{t}\leftarrow\hat{\mathcal{F}}^{t}+\Delta\mathcal{F}^{t}$

return _$\mathcal{F}^{t},\hat{\mathcal{F}}^{t+1}$_

Algorithm 2 Temporal iterative deblurring (TID). Figure 3: Initial flow $\hat{\mathcal{F}}^{t}$, residual flow $\Delta\mathcal{F}^{t}$ and final flow $\mathcal{F}^{t}$ as time progresses and new event bins keeps coming. A trend of increasing quality in $\mathcal{F}^{t}$ and lowering magnitude in $\Delta\mathcal{F}^{t}$ can be observed, implying that the flow is iteratively refined through time. | EPE | AE | 1PE | 2PE | 3PE |  Model Size | GMAC |  Memory Footprint
---|---|---|---|---|---|---|---|---
EV-FlowNet [41] | 2.32 | 7.90 | 55.4 | 29.8 | 18.6 |  14M | 358 |  120MB
E-RAFT [12] | 0.79 | 2.85 | 12.7 | 4.7 | 2.7 |  5.3M | 256 |  156MB
E-RAFT w/o Correlation Volume | 1.20 | 3.75 | 27.5 | 11.5 | 6.3 |  4.5M | 208 |  40MB
Ours (ID, 4 iterations) | 0.88 | 3.21 | 15.6 | 5.8 | 3.2 |  0.9M | 171 |  15MB
Ours (TID, 1 iteration) | 0.93 | 3.98 | 19.3 | 6.6 | 3.4 |  1.2M | 43 |  15MB
Table 1: Evaluation on DSEC-Flow dataset with model statistics. The best
metric is in bold, while the second best is underlined.

###  3.5 Network Architecture

We select a single layer of ConvGRU with 64 feature channels as our backbone
network. The Encoder network consists of 5 convolutional layers with 2 layers
adding a residual connection. Event bins are downsampled to 1/8 of their
original size and expanded to 64 channels before being processed by the
backbone RNN. The WSM module takes the same architecture as the Encoder
network. For details of the architecture of the modules, we refer to the
supplementary material.

###  3.6 Loss

For the update scheme of iterative deblurring (ID), we apply L1 loss between
the final flow and ground truth:

$\mathcal{L}_{\text{ID}}=||\mathcal{F}_{\text{gt}}-\mathcal{F}_{N}||_{1}$ (4)

For training temporal iterative deblurring (TID), we take a prediction
sequence of length $T$ and enforce an L1 loss on both the intermediate current
flows and future flows and weigh them using a geometric series to penalize
error more on later iterations, where $\gamma\in(0,1)$:

$\mathcal{L}_{\text{TID}}=\sum_{t=0}^{T}\gamma^{T-t}(||\mathcal{F}_{\text{gt}}^{t}-\mathcal{F}^{t}||_{1}+||\mathcal{F}_{\text{gt}}^{t+1}-\hat{\mathcal{F}}^{t+1}||_{1})$
(5)

##  4 Experiments

###  4.1 Training Details

We train and evaluate our models on the DSEC Optical Flow dataset [11] [12].
We set the number of deblurring iterations $N$ to be 4 for Algorithm 1 and the
sequence length $T=5$ and $\gamma=0.8$ in Eq. 5. We use an event
representation with $\text{B}=15$ bins for every $100\text{\,}\mathrm{ms}$ of
events. We train our models implemented with PyTorch [29] on NVIDIA RTX 3090
and A100 GPUs. We use Adam [21] optimizer with onecycle learning rate
scheduler [35] under a maximum learning rate of $1\text{\times}{10}^{-4}$ and
a batch size of 3 for training until the validation loss is converged. We
apply random cropping to a size of 288 by 384 and 50% probability of
horizontal flipping and 10% probability of vertical flipping as data
augmentation during training.

###  4.2 Baselines and Metrics

Here we compare against a U-Net baseline EV-Flownet [41] as well as the state-
of-the-art method E-RAFT [12] on the public benchmark. In addition, to compare
our methods against an iteratively refined model without using explicit
correlation volume, we train an adapted version of E-RAFT by removing the
correlation block from its architecture while keeping everything else intact.
The results are presented in Tab. 1. We report five performance metrics as
well as three model statistics:

  * •

EPE (Endpoint error): The average of the L2-Norm of the optical flow error
vector in pixels.

  * •

AE (Angular error): The average of the angular error of the optical flow
vector in degrees.

  * •

nPE (n-pixel outlier percentage): the percentage of pixels with optical flow
magnitude error greater than n pixels. n is either 1, 2, or 3.

  * •

Model Size: The number of learnable parameters of the model in millions of
parameters.

  * •

GMAC: Number of Giga multiply-and-accumulation operations required for
estimating one optical flow during inference. Measured in billions of
operations.

  * •

Memory footprint: the minimum amount of working memory required during
inference while executing the computation graph of the model.

Figure 4: Qualitative results of optical flow predictions on DSEC-Flow with
highlighted regions of interest. Images are for visualization purposes only,
as the optical flow estimates are based on events. Examples taken from the
four test sequences reveal our ID method is close to E-RAFT while
outperforming the adapted version of E-RAFT which operates without correlation
volume.

From Tab. 1, both our ID and TID methods outperform EV-FlowNet and the adapted
E-RAFT without correlation volume. In addition, the ID method achieves
performance metrics close to E-RAFT with a 10% drop in terms of EPE.

However, our methods achieve such performance with only 20% of the parameter
count and 10% of the memory footprint compared to E-RAFT. The significant
savings in memory footprint compared to E-RAFT are contributed by not only the
smaller network but mainly the unnecessity of storing the all-pair correlation
volume required in E-RAFT, which serves as the workhorse behind the
performance of the method. To prove that, the adapted E-RAFT without computing
correlation volume now has a much smaller memory footprint but the performance
has also deteriorated significantly by 50% by the metric of EPE.

While ID achieves better performance, TID is drastically faster by only
iterating once. By leveraging the temporal prior, we have managed to retain
the quality of prediction while incurring a performance drop of only 6%
compared to 4 iterations of ID and 18% compared to the state-of-the-art method
E-RAFT, which consumes almost 6 times more compute. Moreover, with E-RAFT, the
processing of event bins can only begin after all events under the time window
have been accumulated, which incurs large processing latency. Since TID
employs per-bin processing and only iterates the event bins once, it is
possible to parallelize the processing of bins as events arrive in real-time.
In this way, inference latency can be reduced by at least a factor of the
number of bins to be processed.

###  4.3 Qualitative Results

Fig. 4 presents examples of optical flow predictions taken from different test
sequences of DSEC-Flow. We observe that our methods are able to recover the
details of small and fast-moving objects such as the road poles in the
interlaken_01_a sequence, which proves to be challenging for the correlation-
free E-RAFT. Our methods are also successful at discerning moving objects
which are close to each other shown in the thun_01_b sequence, whereas the
adapted E-RAFT fails to separate the event streams from the further away pole
with the closer pole, leading to a blurry optical flow field in the space
between the objects. We also note that adapted E-RAFT consistently leaves a
trailing edge for close objects moving out of the view, as shown in the
zurich_city_14_c sequence, indicating the model is ignorant of the temporal
information of event bins which would otherwise be explicitly captured in the
correlation volumes.

In terms of limitations, our methods produce less sharp motion boundaries
compared to E-RAFT, which is where most of the difference in predictions
occurs, as shown in the difference map provided in the supplementary material.
This does not seem to be caused by the lack of correlation volume since the
adapted E-RAFT also preserves sharper motion boundaries. We speculate this is
due to our recurrent processing of event bins as the RNN incurs forgetting of
information from earlier bins. We also note our methods fail to discern tiny
objects undergoing small motion such as the region shown in the
zurich_city_15_a sequence, as does the adapted E-RAFT without correlation
volumes.

In summary, our proposed IDNet achieves comparable performance against the
state of the art where the majority of the quality degradation occurs at
motion boundaries. Our method also outperforms a baseline without correlation
volumes, confirming our iteratively deblurring principle as a lightweight yet
effective alternative to processing correlation volumes in optical flow
estimation.

###  4.4 Ablation Studies

| EPE | AE | 1PE | 3PE
---|---|---|---|---
ID 4 iters | 0.88 | 3.21 | 15.6 | 3.2
ID 2 iters | 0.94 | 3.43 | 18.1 | 3.7
ID 1 iter | 1.30 | 4.82 | 33.7 | 6.7
ID 4 iter w/o deblurring | 1.24 | 4.49 | 31.1 | 6.6
ID 4 iters w/o WSM | 1.02 | 4.07 | 20.8 | 4.0
Table 2: Ablation study on DSEC-Flow dataset w.r.t the effectiveness of
iterative deblurring.

#### Iterative refinement and Deblurring

We study the impact of introducing iterative deblurring to our model. The
results are reported in Tab. 2. For this experiment, we first set the number
of deblurring iterations down to 1. By neither performing iterative processing
nor deblurring, the performance dropped by 47% on EPE. Then we introduce the
recurrency by setting the number of iterations back to 4 but without
performing deblurring. That is, the network would take raw event bins instead
of the deblurred ones as outlined in Algorithm 1. Still, a 41% decrease in EPE
is observed, demonstrating the effectiveness of deblurring as a good inductive
bias in our proposed model. Lastly, we remove the warm starting module so that
the memory of the backbone RNN would be initialized to zero for all
iterations. This time, EPE and AE went up by roughly 16%, revealing the
recurrency is better complemented with an informative prior provided to the
backbone RNN.

We also study the impact of the number of deblurring iterations on the
performance by comparing ID models with $N=1,2,4$ iterations. We observe that
iterating twice already lowers EPE by 27%, while two extra deblurring
iterations bring a diminishing further improvement of 5%.

#### Forward Propagation Methods

Figure 5: Comparison of optical flow propagation schemes. The flow prediction made by direct transport from E-RAFT [12] suffers from artifacts due to occlusion while our learning-based approach yields a smooth optical flow field with lower EPE error. | EPE | AE
---|---|---
TID w/ Direct Transport | 1.25 | 4.60
TID w/ Learning | 0.93 | 3.98
E-RAFT w/ Direct Transport | 0.79 | 2.85
E-RAFT w/ Learning | - | -
Table 3: Impact of direct vs learned forward propagation schemes on IDNet-TID
and E-RAFT.

Previous methods such as E-RAFT[12] use warping to warm start the prediction
for the next timestep, which simply transports the optical flow vector
estimated from the current timestep by itself to arrive at a coarse estimate
for the next timestep. However, we note that even with state-of-the-art flow
predictions made by E-RAFT, directly transporting those will lead to artifacts
shown in Fig. 5, such as the white regions with zero flow caused by occlusion.
Instead, we propose to learn future flow directly as described in Algorithm 2.
This approach increases the accuracy of future flow prediction compared to the
direct transport method, and when combined with TID, gives a significant 25%
boost in the performance on DSEC-Flow, as shown in Tab. 3. Interestingly,
replacing the direct transport scheme with a learning-based prediction on
E-RAFT does not lead to effective training. We speculate this is due to the
intended design of its update module to specialize only in predicting the
current optical flow.

##  5 Conclusion

In this work, we propose IDNet, a lightweight yet high-performing event-based
optical flow method. Our algorithm takes the idea of using correlation for
flow estimation, the working element in the state-of-the-art method, and
implements it implicitly through iterative deblurring. The resulting model is
much smaller in size and alleviates the huge memory footprint of storing
correlation volume, while achieving a performance close to the state of the
art on the public benchmark. Through ablation studies, we show the
effectiveness of iterative deblurring in estimating flow from event-based data
and encourage the community to explore such a combination for other
architectures and vision tasks.

## Acknowledgements

This work was sponsored by the Office of Naval Research (ONR) Global under
grant number N629092112014. The views and conclusions contained herein are
those of the authors only and should not be interpreted as representing those
of the U.S. Government.

## References

  * [1] Patrick Bardow, Andrew J. Davison, and Stefan Leutenegger.  Simultaneous optical flow and intensity estimation from an event camera.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 884–892, 2016.
  * [2] Ryad Benosman, Charles Clercq, Xavier Lagorce, Sio-Hoi Ieng, and Chiara Bartolozzi.  Event-based visual flow.  IEEE Transactions on Neural Networks and Learning Systems, 25(2):407–417, 2014.
  * [3] Ryad Benosman, Sio-Hoi Ieng, Charles Clercq, Chiara Bartolozzi, and Mandyam Srinivasan.  Asynchronous frameless event-based optical flow.  Neural Networks, 27:32–37, 2012.
  * [4] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck.  A 240 × 180 130 db 3 µs latency global shutter spatiotemporal vision sensor.  IEEE Journal of Solid-State Circuits, 49(10):2333–2341, 2014.
  * [5] Haiyang Chao, Yu Gu, and Marcello Napolitano.  A survey of optical flow techniques for robotics navigation applications.  Journal of Intelligent & Robotic Systems, 73(1):361–372, 2014\.
  * [6] Ziluo Ding, Rui Zhao, Jiyuan Zhang, Tianxiao Gao, Ruiqin Xiong, Zhaofei Yu, and Tiejun Huang.  Spatio-temporal recurrent networks for event-based optical flow estimation.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 36, pages 525–533, 2022.
  * [7] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2758–2766, 2015.
  * [8] G. Gallego, T. Delbruck, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis, and D. Scaramuzza.  Event-based vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(01):154–180, 1 2022.
  * [9] Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza.  A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3867–3876, 2018.
  * [10] Daniel Gehrig, Mathias Gehrig, Javier Hidalgo-Carrio, and Davide Scaramuzza.  Video to events: Recycling video datasets for event cameras.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6 2020.
  * [11] Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza.  Dsec: A stereo event camera dataset for driving scenarios.  IEEE Robotics and Automation Letters, 6(3):4947–4954, 2021.
  * [12] Mathias Gehrig, Mario Millhäusler, Daniel Gehrig, and Davide Scaramuzza.  E-raft: Dense optical flow from event cameras.  In International Conference on 3D Vision (3DV), 2021.
  * [13] Jesse Hagenaars, Federico Paredes-Valles, and Guido de Croon.  Self-supervised learning of event-based optical flow with spiking neural networks.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7167–7179. Curran Associates, Inc., 2021.
  * [14] Berthold K.P. Horn and Brian G. Schunck.  Determining optical flow.  Artificial Intelligence, 17(1):185–203, 1981.
  * [15] Liwen Hu, Rui Zhao, Ziluo Ding, Lei Ma, Boxin Shi, Ruiqin Xiong, and Tiejun Huang.  Optical flow estimation for spiking camera.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6 2022.
  * [16] Y Hu, S C Liu, and T Delbruck.  v2e: From video frames to realistic DVS events.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2021.
  * [17] Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy.  LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8981–8989, 2018.
  * [18] Junhwa Hur and Stefan Roth.  Iterative residual refinement for joint optical flow and occlusion estimation.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5747–5756, 2019.
  * [19] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017.
  * [20] Huaizu Jiang, Deqing Sun, Varan Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1 2018.
  * [21] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [22] Chankyu Lee, Adarsh Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy.  Spike-flownet: Event-based optical flow estimation with energy-efficient hybrid neural networks.  In European Conference on Computer Vision, pages 366–382. Springer, 2020.
  * [23] Zhuoyan Li, Jiawei Shen, and Ruitao Liu.  A lightweight network to learn optical flow from event data.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 1–7, 2021.
  * [24] Guo Lu, Xiaoyun Zhang, Wanli Ouyang, Li Chen, Zhiyong Gao, and Dong Xu.  An end-to-end learning framework for video compression.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3292–3308, 2021.
  * [25] Yao Lu, Jack Valmadre, Heng Wang, Juho Kannala, Mehrtash Harandi, and Philip H. S. Torr.  Devon: Deformable volume network for learning optical flow.  2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Mar 2020.
  * [26] Bruce D. Lucas and Takeo Kanade.  An iterative image registration technique with an application to stereo vision.  In Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’81, page 674–679, San Francisco, CA, USA, 1981. Morgan Kaufmann Publishers Inc.
  * [27] Elias Mueggler, Christian Forster, Nathan Baumli, Guillermo Gallego, and Davide Scaramuzza.  Lifetime estimation of events from dynamic vision sensors.  In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 4874–4881, 2015.
  * [28] Federico Paredes-Valles and Guido C. H. E. de Croon.  Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3446–3455, June 2021.
  * [29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library, 2019\.
  * [30] Wachirawit Ponghiran, Chamika Mihiranga Liyanagedera, and Kaushik Roy.  Event-based temporally dense optical flow estimation with sequential neural networks, 2022.
  * [31] Tong Qin, Peiliang Li, and Shaojie Shen.  Vins-mono: A robust and versatile monocular visual-inertial state estimator.  IEEE Transactions on Robotics, 34(4):1004–1020, 2018.
  * [32] Anurag Ranjan and Michael J. Black.  Optical flow estimation using a spatial pyramid network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [33] Rajesh P. N. Rao and Dana H. Ballard.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.  Nature Neuroscience, 2(1):79–87, 1999.
  * [34] Shintaro Shiba, Yoshimitsu Aoki, and Guillermo Gallego.  Secrets of event-based optical flow.  In European Conference on Computer Vision (ECCV), 2022.
  * [35] Leslie N. Smith and Nicholay Topin.  Super-convergence: very fast training of neural networks using large learning rates.  Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, May 2019.
  * [36] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018.
  * [37] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 402–419, Cham, 2020. Springer International Publishing.
  * [38] Longguang Wang, Yulan Guo, Li Liu, Zaiping Lin, Xinpu Deng, and Wei An.  Deep video super-resolution using hr optical flow estimation.  IEEE Transactions on Image Processing, 29:4323–4336, 2020.
  * [39] Jian Xiong, Hongliang Li, Qingbo Wu, and Fanman Meng.  A fast hevc inter cu selection method based on pyramid motion divergence.  IEEE Transactions on Multimedia, 16(2):559–564, 2014.
  * [40] Chengxi Ye, Anton Mitrokhin, Cornelia Fermüller, James A. Yorke, and Yiannis Aloimonos.  Unsupervised learning of dense optical flow, depth and egomotion with event-based sensors.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5831–5838, 2020.
  * [41] Alex Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis.  Ev-flownet: Self-supervised optical flow estimation for event-based cameras.  Robotics: Science and Systems XIV, Jun 2018.
  * [42] Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Daniilidis.  The multivehicle stereo event camera dataset: An event camera dataset for 3d perception.  IEEE Robotics and Automation Letters, 3(3):2032–2039, 2018.
  * [43] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis.  Unsupervised event-based learning of optical flow, depth, and egomotion.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6 2019.