# Tails: Chasing Comets with the Zwicky Transient Facility and Deep Learning

Dmitry A. Duev Division of Physics, Mathematics, and Astronomy, California
Institute of Technology, Pasadena, CA 91125, USA  Bryce T. Bolin Division of
Physics, Mathematics, and Astronomy, California Institute of Technology,
Pasadena, CA 91125, USA  IPAC, California Institute of Technology, 1200 E.
California Blvd, Pasadena, CA 91125, USA  Matthew J. Graham Division of
Physics, Mathematics, and Astronomy, California Institute of Technology,
Pasadena, CA 91125, USA  Michael S. P. Kelley Department of Astronomy,
University of Maryland, College Park, MD 20742, USA  Ashish Mahabal Division
of Physics, Mathematics, and Astronomy, California Institute of Technology,
Pasadena, CA 91125, USA  Center for Data Driven Discovery, California
Institute of Technology, Pasadena, CA 91125, USA  Eric C. Bellm DIRAC
Institute, Department of Astronomy, University of Washington, 3910 15th Avenue
NE, Seattle, WA 98195, USA  Michael W. Coughlin School of Physics and
Astronomy, University of Minnesota, Minneapolis, Minnesota 55455, USA  Richard
Dekany Caltech Optical Observatories, California Institute of Technology,
Pasadena, CA 91125, USA  George Helou IPAC, California Institute of
Technology, 1200 E. California Blvd, Pasadena, CA 91125, USA  Shrinivas R.
Kulkarni Division of Physics, Mathematics, and Astronomy, California Institute
of Technology, Pasadena, CA 91125, USA  Frank J. Masci IPAC, California
Institute of Technology, 1200 E. California Blvd, Pasadena, CA 91125, USA
Thomas A. Prince Division of Physics, Mathematics, and Astronomy, California
Institute of Technology, Pasadena, CA 91125, USA  Reed Riddle Caltech Optical
Observatories, California Institute of Technology, Pasadena, CA 91125, USA
Maayane T. Soumagnac Lawrence Berkeley National Laboratory, 1 Cyclotron Road,
Berkeley, CA 94720, USA  Department of Particle Physics and Astrophysics,
Weizmann Institute of Science, Rehovot 76100, Israel  Stéfan J. van der Walt
Berkeley Institute for Data Science, University of California, Berkeley,
Berkeley, CA 94720, USA

(Received January 11, 2021; Revised February 23, 2021; Accepted February 25,
2021)

###### Abstract

We present Tails, an open-source deep-learning framework for the
identification and localization of comets in the image data of the Zwicky
Transient Facility (ZTF), a robotic optical time-domain survey currently in
operation at the Palomar Observatory in California, USA. Tails employs a
custom EfficientDet-based architecture and is capable of finding comets in
single images in near real time, rather than requiring multiple epochs as with
traditional methods. The system achieves state-of-the-art performance with 99%
recall, 0.01% false positive rate, and 1-2 pixel root mean square error in the
predicted position. We report the initial results of the Tails efficiency
evaluation in a production setting on the data of the ZTF Twilight survey,
including the first AI-assisted discovery of a comet (C/2020 T2) and the
recovery of a comet (P/2016 J3 = P/2021 A3).

astroinformatics — astronomy data analysis — convolutional neural networks —
comets — surveys

††journal: AJ††facilities: PO:1.2m, ZTF††software: astropy (Astropy
Collaboration et al., 2018), Fritz (https://github.com/fritz-marshal/fritz),
Kowalski (Duev et al., 2019), matplotlib (Hunter, 2007), numpy (Harris et al.,
2020), pandas (pandas development team, 2020), pypride (Duev et al., 2016),
SEP (Barbary, 2016), sbpy (Mommert et al., 2019), TensorFlow (Abadi et al.,
2016), ZChecker (Kelley et al., 2019)

##  1 Introduction

Comets have mesmerized humans for millennia, frequently offering, arguably,
some of the most spectacular sights in the night sky. Containing the original
materials from when the Solar System first formed, comets provide a unique
insight into the distant past of our Solar System. The recent discovery of the
first interstellar comet 2I/Borisov by amateur astronomer Gennadiy Borisov
predictably sparked much excitement and enthusiasm among astronomers and the
general public alike (e.g., Bolin et al., 2020; Fitzsimmons et al., 2019;
Guzik et al., 2020). Such objects could potentially provide important
information on the formation of other stellar systems. It is a very exciting
time to look for comets: the large-scale time-domain surveys that are
currently in operation, such as the ZTF (Bellm et al., 2019a; Graham et al.,
2019), Pan-STARRS (Chambers et al., 2016), or ATLAS (Tonry et al., 2018), and
the upcoming ones such as BlackGEM (Bloemen et al., 2016) and Vera Rubin
Observatory / LSST (Ivezić et al., 2008) offer the richest data sets ever
available to mine for comets.

Traditional comet detection algorithms rely on multiple observations of
cometary objects that are linked together and used to fit an orbital solution.
To the best of our knowledge, the previous attempts to take the comet’s
morphology in the optical image data into consideration in the detection
algorithms have not led to reliable and robust results.

In this work, we present Tails – a state-of-the-art deep-learning-based system
for the identification and localization of comets in the image data of ZTF.
Tails employs an EfficientDet-based architecture (Tan et al., 2019) and is
thus capable of finding comets in single images in near real time, rather than
requiring multiple epochs as with traditional methods.

The Tails’ code is open-source and can be found in the “dmitryduev/tails”
repository on GitHub. The version of the code aligned with this publication is
archived on Zenodo at 10.5281/zenodo.4563226.

###  1.1 The Zwicky Transient Facility

The Zwicky Transient Facility (ZTF)111https://ztf.caltech.edu is a state-of-
the-art robotic time-domain sky survey capable of visiting the entire visible
sky north of $-30^{\circ}$ declination every night. ZTF observes the sky in
the $g$, $r$, and $i$ bands at different cadences depending on the scientific
program and sky region (Bellm et al., 2019a; Graham et al., 2019). The 576
megapixel camera with a 47 deg2 field of view, installed on the Samuel Oschin
48-inch (1.2-m) Schmidt Telescope, can scan more than 3750 deg2 per hour, to a
$5\sigma$ detection limit of 20.7 mag in the $r$ band with a 30-second
exposure during new moon (Masci et al., 2019a; Dekany et al., 2020).

The ZTF Partnership has been running a specialized survey, the Twilight Survey
(ZTF-TS) that operates at Solar elongations down to 35 degrees with an
$r$-band limiting magnitude of 19.5 (Ye et al., 2020; Bellm et al., 2019b).
ZTF-TS has so far resulted in the discovery of a number of Atira asteroids
(orbits interior to the Earth’s) as well as the first inner-Venus object, 2020
AV2 (Ip et al., 2020). Motivated by the success, ZTF-TS will be expanded in
Phase II of the project, which commenced in December 2020.

Comets become more easily detectable when close to the Sun as they become
brighter and start exhibiting more pronounced coma and tails. Furthermore, it
has been shown that the most detectable direction of approach of an
interstellar object is from directly behind the Sun because of observational
selection effects (Jedicke et al., 2016) and the fact that this direction has
a greater cross section for asteroids to bend around and pass into the
visibility volume (Engelhardt et al., 2017; Do et al., 2018).

Tails automates the search for comets with detectable morphology. While
trained and evaluated on a large corpus of ZTF data, in this work we focus on
Tails’ performance when applied to the ZTF-TS data.

##  2 Tails: a deep learning framework for the identification and localization
of comets

Deep learning (DL) is a subset of machine learning that employs artificial
many-layer neural networks (McCulloch & Pitts, 1943). DL systems are able to
discover, in a highly automated manner, efficient representations of the data,
simplifying the task of finding the meaningful sought-after patterns in them.
We refer the reader to a brilliant introduction into DL given in Géron (2019).

DL systems often reach near-optimal performance for a given task and are able
to learn even very complicated, highly non-linear mappings between the input
and output spaces. The art of building applied DL systems involves two major
challenges: finding a suitable network architecture and, more importantly,
constructing a large, labeled, representative data set for the network
training. In the case of comet detection, the training set must reflect the
possible variations across different seeing conditions, filters, sky location,
CCDs, and include data artifacts caused by, for example, cross-talk or
telescope reflections.

###  2.1 Data set

Figure 1: Distribution of over 60,000 individual observations of comets as a
function of the predicted total magnitude (as reported by JPL Horizons) used
in the seed sample.

(a) 114P/Wiseman–Skiff observed on 2019/10/19

(b) 29P/Schwassmann–Wachmann observed on 2019/10/19

(c) C/2019 D1 (Flewelling) observed on 2019/07/22

(d) 260P/McNaught observed on 2019/10/19

Figure 2: Comet examples from the training set. The first image in a triplet
is the epochal science image, the second – the reference image, the third –
the ZOGY difference image.

To build a seed sample for labeling, we first identified all potential
observations of known comets conducted with ZTF from March 5, 2018 - March 4,
2020, based on their predicted position and brightness. The code for
accomplishing that is based on the Python libraries pypride (Duev et al.,
2016) and solarsyslib (Jensen-Clem et al., 2018) and uses the comet
ephemerides obtained from the Minor Planet Center
(MPC)222https://minorplanetcenter.net/iau/MPCORB/CometEls.txt for a coarse
search, followed by a JPL Horizons333https://ssd.jpl.nasa.gov/horizons.cgi
(Giorgini et al., 1996) query for precision.

To provide more contextual information, epochal image data are supplemented by
properly aligned reference images of the corresponding patches of sky and
difference (epochal minus reference) images generated with the ZOGY algorithm
(Zackay et al., 2016), all produced by the ZTF Science Data System at
Caltech’s IPAC (Masci et al., 2019a). Finally, we generate image triplet
cutouts of size 256 by 256 pixel, which in angular measure translates into
$4\arcmin.3$ by $4\arcmin.3$ at ZTF’s pixel scale of $1\arcsec.01/pix$.

We selected over 60,000 individual observations with the total comet magnitude
ranging from 10 to 23 (as reported by JPL Horizons; see Fig. 1), out of which
about 20,000 were sourced for manual annotation. This resulted in an initial
sample of 3,000 examples with identifiable morphology.

We also compiled a set of approximately 20,000 negative examples consisting of
point-like cometary detections, patches of sky with no identified transient or
variable sources, CCD-edge cases, and a wide range of real (point-source)
transient and bogus (e.g. artifacts due to bright stars, optical ghosts and
“dementors”) samples from the Braai data set (Duev et al., 2019).

To expand the data set, we then assembled a standard ResNet-based (He et al.,
2015) classifier for comet identification. With this basic classification
model, we ran several rounds of an active-learning-like procedure, where we
would first train the classifier, evaluate it on the whole data set, sample
both confident predictions and the cases close to the classifier’s decision
boundary, manually inspect and label those examples and add them to the
training set. Roughly 2,000 positive and 2,000 negative examples were added to
the training set via this method.

The resulting training data set contains about 5,000 positive and 22,000
negative examples (see Fig. 2). Each triplet in the set has been assigned a
label $[p_{c},x,y]$, where $p_{c}$ marks the presence of a comet in the image
and $x,y\in[0,1]$ is the relative positions of the comet’s “center of mass”,
as reported by JPL Horizons. For positive examples this translates into
$[1,x_{JPL},y_{JPL}]$, for negative ones – $[0,?,?]$, where question marks
mean that these do not affect the loss in this case.

###  2.2 Deep neural network architecture and training

Figure 3: Tails architecture: a custom EfficientDet D0-based network (Tan et
al., 2019). A batch of duplet or triplet image stacks of size
$(n_{b},256,256,[2|3])$, correspondingly, is passed through an EfficientNet B0
backbone, where $n_{b}$ is the number of stacks in the batch. The extracted
features from the last five blocks/levels of the backbone network are passed
through a bidirectional feature-pyramid network (BiFPN). The resulting five
output tensors denoted in colored circles are fed into the head network, which
outputs the probability of the image containing a comet $p_{c}$ and its
predicted relative position $(x,y)$.

Tails adopts a custom architecture (see Fig. 3) based on EfficientDet D0 (Tan
et al., 2019), a variant of a state-of-the-art architecture designed for
object detection - a computer vision technique for the identification and
location of objects in image data.

This architecture delivers best-in-class object detection efficiency and
performance across a wide range of resource constraints. This is achieved by
using EfficientNet – state-of-the art backbone networks for feature
extraction, a weighted bi-directional feature pyramid network (BiFPN), which
allows easy and fast multi-scale feature fusion, and a compound scaling method
that simultaneously and uniformly scales the resolution, depth, and width for
all backbone, feature, and location/class prediction networks (Tan et al.,
2019).

The use of a BiFPN, which effectively represents and processes multi-scale
features, makes this architecture particularly well-suited for the problem of
morphology-based comet identification and localization.

A batch of triplet image stacks of size $(n_{b},256,256,3)$, where $n_{b}$ is
the number of stacks in the batch, is passed through an EfficientNet B0
backbone (Tan & Le, 2019). The extracted features from the last five
blocks/levels of the network are passed through the BiFPN. The resulting five
output tensors denoted in colored circles in Fig. 3 are fed into the head
network, which outputs the probability of the image containing a comet $p_{c}$
and its centroid’s predicted relative $(x,y)$ position444We note that standard
object detection algorithms typically output bounding boxes and corresponding
object class probabilities, i.e. sets of ($4+n_{classes}$) numbers. Our
approach allowed us to simplify the head network architecture and both
simplify and speed up the assembly of the training data set, bypassing the
unnecessary complexity and potential inaccuracy of drawing bounding boxes
around known comet detections..

We defined the loss function as follows:

$L=w_{c}\cdot L_{c}+w_{p}\cdot L_{p}$ (1)

where $L_{c}$ denotes the binary cross-entropy function for the label $c$ (1 –
there is a comet in the image, 0 – there is no comet) and the predicted
probability $p_{c}$. If $\left\lfloor p_{c}\right\rceil=1$, $L_{p}$ is
computed as an $L_{1}$ loss for the relative position $(x,y)$ and its
prediction $(x_{p},y_{p})$ with a small $L_{2}$ regularizing term (with
$\epsilon=10^{-3}$), and $w_{c}$ and $w_{p}$ denote the weights of the two
terms, respectively:

$\begin{split}L_{c}=\sum
c\cdot\mathrm{log}(p_{c})+(1-c)\cdot\mathrm{log}(1-p_{c})\\\
L_{p}=\sum\left\lfloor p_{c}\right\rceil\cdot\Big{(}|x-x_{p}|+|y-y_{p}|+\\\
\epsilon\cdot\sqrt{|x-x_{p}|^{2}+|y-y_{p}|^{2}}\Big{)}\end{split}$ (2)

We employed the Adam optimizer (Kingma & Ba, 2014), a batch size of 32, and a
81%/9%/10% training/validation/test data split. For data augmentation, we
applied random horizontal and vertical flips of the input data; no random
rotations and translations were added. We note that the test/validation sets
did not contain augmented data from the training set. We used standard
techniques to maximize training performance: if no improvement in validation
loss was observed for 10 epochs, the learning rate was reduced by a factor of
2, and training was stopped early if no improvement was observed for 30
epochs.

The EfficientNet’s weights were randomly initialized555We experimented with
pre-trained weights, however that neither helped the network to reach
convergence faster, nor did it affect the final performance. We believe this
is likely due to the fact that astronomical images are very different from
those in commonly-used data sets.. We first set $w_{c}=10,w_{p}=1$ to allow
for a fast convergence of the feature-extracting part of the network. To fine-
tune the performance, we trained Tails on a balanced data set setting
$w_{c}=1.1,w_{p}=1$ and monitored the validation loss for early stopping, then
bumped $w_{p}=2$ and monitored the validation positional RMSE; finally, added
the omitted negative examples and again monitored the validation loss for
early stopping.

The resulting classifiers were put through the same active-learning-like
procedure as was employed in the initial data set assembly, using several
months of ZTF Twilight survey data.

##  3 Tails performance

Evaluated on the test set, with a score $p_{c}$ threshold of 0.5, Tails
demonstrates false positive and false negative rates (FPR and FNR) of 1.7%,
and a $\sim 1-2$ pixel median RMSE of the predicted comet “centroid” position
versus that acquired from JPL Horizon (see Fig. 4).

(a) False positive rate (FPR) and false negative rate (FNR) as a function of
score $p_{c}$. FPR and FNR balance out at around 1.7% for a score threshold of
0.5.

(b) RMSE of the predicted comet position versus the reported on JPL Horizon
for the 650 positive samples from the test set.

Figure 4: Test set performance of Tails. The set contains 1,400 negative and
650 positive examples.

The ZTF instrument’s CCD mosaic has 16 individual 6k $\times$ 6k science CCDs.
The raw ZTF image data are split into four readout quadrants per CCD and all
processing is conducted independently on each CCD readout quadrant. We
tessellate each 3k $\times$ 3k CCD-quadrant image into a 13 $\times$ 13 grid
of overlapping 256 $\times$ 256 pixels tiles and evaluate Tails on
those.666Standard fully-convolutional approaches often used in computer vision
proved to be an overkill in this case.

(a) Candidate scanning page. The users can inspect the candidates and save
vetted objects to one or more groups. Candidates that are not saved to any
group within 7 days are removed from Fritz.

(b) Source page. It aggregates and displays in an interactive manner all kinds
of information related to an object that exists on Fritz, such as photometry,
spectroscopy, auto-annotations, comments, finder charts, follow-up requests,
and other data.

Figure 5: Screenshots of the Fritz user interfaces used for Tails candidate
inspection and vetting.

Tails has been deployed in production since late June 2020. We have
implemented a “sentinel” service777See https://github.com/dmitryduev/tails
that processes the incoming data in real time and posts the plausible
candidates to Fritz888https://github.com/fritz-marshal/fritz, the ZTF Phase II
open-source science data platform (van der Walt et al., 2019; Duev et al.,
2019; Kasliwal et al., 2019), for further manual inspection and vetting. The
candidates are auto-annotated with the detailed information on the detection
such as the score, CCD and sky positions, and cross-matches with known Solar
System objects. Fig. 5 shows screenshots of the Fritz user interfaces used in
the process.

It takes about 5 hours to run inference on a typical set of nightly ZTF
Twilight data ($\sim 45$ 30-second exposures) on an e2-highcpu-32 virtual
machine instance (32 vCPU, 32 GB memory, SSD disk) on the Google Cloud
Platform, including I/O operations.

Consistently with the expected rate of comet observations, a typical run on
nightly Twilight data yields a few dozen candidates, which, given the typical
number of processed tiles, gives an empirical false positive rate (FPR) value
of about 0.01%.

The scanning results are accumulated and used to expand the training set and
improve Tails’ performance.

We have evaluated Tails’ performance on a random sample of 200 observations of
known comets with identifiable morphology in July-August 2020 and found an
empirical recall value of 99%.

Fig. 6 shows a number of comet candidates not from the training set identified
by Tails, including some of the ZTF observations of the comet 2I/Borisov.
Optical artifacts resembling cometary objects are the main source of
contamination.

(a) 2I/Borisov observed on 2019/10/05

(b) 2I/Borisov observed on 2019/10/15

(c) C/2010 U3 (Boattini) observed on 2020/08/11

(d) Bogus detection next to a bright star

(e) Bogus detection due to a brightened satellite trail

(f) Bogus detection due to a telescope reflection

Figure 6: Candidates identified by Tails. Panels (a), (b), and (c) on the left
show the detections of real comets. Typical false positives are shown on
Panels (d), (e), and (f) on the right. For each image triplet, the left pane
shows the epochal science exposure, the middle pane – the reference image of
the corresponding patch of sky, and the right pane – the ZOGY difference
image.

###  3.1 Discovery of comet C/2020 T2

On October 7, 2020, Tails discovered a candidate that was posted to MPC’s
Possible Comet Confirmation Page (PCCP)
999https://minorplanetcenter.net/iau/NEO/pccp_tabular.html as ZTFDD01 (see
Fig. 8). It was later confirmed to be a long-period comet and designated
C/2020 T2 (Palomar), marking the first DL-assisted comet discovery (Duev,
2020). The candidate was found in the Twilight survey data; it was at 19.3 mag
in the ZTF $r$ band. The FWHM of the object was approximately
$2\arcsec.5$–$3\arcsec$, compared to nearby background stars that have FWHM of
$\sim 2\arcsec$. The object showed a tail extending up to $5\arcsec$ in the
westward direction. Table 1 summarizes the orbital elements of C/2020 T2
provided by the MPC and Fig. 7 shows its orbit as of the discovery date.

element | value
---|---
e | 0.9934213
Incl. | 27.87307
Peri. | 150.38279
Node | 83.04834
q | 2.0546940
T | 2021 July 11.14638 TT
Table 1: Orbital elements of C/2020 T2 provided by the MPC. Figure 7: The
orbit of comet C/2020 T2 as of October 7, 2020. Image credit: NASA/JPL-Caltech
/ D. Duev. Figure 8: Discovery image of C/2020 T2 (Palomar), the first DL-
assisted comet discovery by Tails. Taken on October 7, 2020 with the ZTF
camera on the 48-inch Schmidt telescope at Palomar. The left pane shows the
epochal science exposure (256x256 pix cutout), the middle pane – the reference
image, the right pane – the ZOGY difference image. East is to the left, north
is down.

To determine if Tails could have discovered C/2020 T2 before 2020 October 7,
we searched the ZTF archive for all Twilight Survey data covering the
ephemeris position of the comet with the ZChecker software (Kelley et al.,
2019). Eleven nights of data were found between 2020 June 11–20 (evening
twilight) and October 7–21 (morning twilight). The comet was in conjunction
with the Sun between the two sets, and not observable by ZTF. We measured the
brightness of the coma in 4-pix radius apertures, and aperture corrected the
photometry according to the ZTF pipeline documentation. The data are shown in
Fig. 9. Typical seeing was $2\arcsec$ in June, the comet was very faint
($r$=20.2 mag), near the single-image detection limit ($r$=20.4–20.9 mag,
5$\sigma$ point source), and had no morphological features for Tails to pick
up. Thus October 7, 2020 was really the first opportunity for Tails to
discover the comet.

Figure 9: Photometry of comet C/2020 T2 (Palomar) derived from ZTF-TS images
($r$-band) versus time from perihelion. A best-fit model lightcurve is also
shown: $r=9.85+9.54\log_{10}(r_{\rm{h}})+5\log_{10}(\Delta)-\Phi(\theta)$,
where $r_{\rm{h}}$ is the heliocentric distance in au, $\Delta$ is the comet-
observer distance in au, and $\Phi(\theta))$ is the phase angle correction
from Schleicher et al. (1998). $T_{p}$ denotes the time of perihelion passage
(July 11, 2021).

###  3.2 Recovery of comet P/2016 J3 = P/2021 A3 (STEREO)

A comet candidate was identified by a combination of Tails and the ZTF Moving
Object Detection Engine (Masci et al., 2019b) on 2020 January 04 UTC and
submitted to the PCCP as ZTF0Ion (see Fig. 10). It was later identified as a
recovery of comet P/2016 J3 (STEREO) and given the designation P/2016 J3 =
P/2021 A3 (STEREO) (Bolin, 2021). P/2021 A3 was identified in the evening
Twilight survey data at $r$=19.3 mag with a clearly-extended appearance
scoring 0.9 with a coma $\sim$10″ wide and a tail extending past 20″in the
north east direction.

Figure 10: Recovery detection of P/2021 A3 (STEREO) identified by Tails in ZTF
r-band data taken on January 4, 2021. The left pane shows the epochal science
exposure (256x256 pix cutout), the middle pane – the reference image, the
right pane – the ZOGY difference image. East is to the left, north is down.

##  4 Discussion

This work demonstrates the potential of the state-of-the-art deep-learning
computer-vision architecture designs when applied to the problem of
astronomical source detection and localization, with a specific focus on
comets.

We experimented with the input data and trained a version of Tails that
instead of triplet image stacks uses duplets – epochal/reference images,
omitting the ZOGY difference images. Our tests show that this version achieves
essentially the same performance as the one trained on triplets without
requiring image differencing, expanding the range of potential use cases of
Tails.

While Tails is trained only on ZTF data, with transfer learning, it can be
adapted to other sky surveys, including the upcoming Vera Rubin Observatory’s
Legacy Survey of Space and Time (LSST) (Ivezić et al., 2008).

D. A. Duev would like to thank Ivan Duev for assistance with data labeling. D.
A. Duev acknowledges support from Google Cloud and from the Heising-Simons
Foundation under Grant No. 12540303. Based on observations obtained with the
Samuel Oschin Telescope 48-inch and the 60-inch Telescope at the Palomar
Observatory as part of the Zwicky Transient Facility project. ZTF is supported
by the National Science Foundation under Grant No. AST-1440341 and a
collaboration including Caltech, IPAC, the Weizmann Institute for Science, the
Oskar Klein Center at Stockholm University, the University of Maryland, the
University of Washington, Deutsches Elektronen-Synchrotron and Humboldt
University, Los Alamos National Laboratories, the TANGO Consortium of Taiwan,
the University of Wisconsin at Milwaukee, and Lawrence Berkeley National
Laboratories. Operations are conducted by COO, IPAC, and UW. This research has
made use of data and/or services provided by the International Astronomical
Union’s Minor Planet Center. The authors would like to express gratitude to
the anonymous referee.

## References

  * Abadi et al. (2016) Abadi, M., Agarwal, A., Barham, P., et al. 2016, arXiv e-prints, arXiv:1603.04467.  https://arxiv.org/abs/1603.04467
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Barbary (2016) Barbary, K. 2016, Journal of Open Source Software, 1(6), 58, doi: 10.21105/joss.00058
  * Bellm et al. (2019a) Bellm, E. C., Kulkarni, S. R., Graham, M. J., et al. 2019a, PASP, 131, 018002, doi: 10.1088/1538-3873/aaecbe
  * Bellm et al. (2019b) Bellm, E. C., Kulkarni, S. R., Barlow, T., et al. 2019b, PASP, 131, 068003, doi: 10.1088/1538-3873/ab0c2a
  * Bloemen et al. (2016) Bloemen, S., Groot, P., Woudt, P., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9906, Ground-based and Airborne Telescopes VI, 990664, doi: 10.1117/12.2232522
  * Bolin et al. (2020) Bolin, B. T., Lisse, C. M., Kasliwal, M. M., et al. 2020, AJ, 160, 26, doi: 10.3847/1538-3881/ab9305
  * Bolin (2021) Bolin, B. T. e. a. 2021, Minor Planet Electronic Circulars, 2021-A157
  * Chambers et al. (2016) Chambers, K. C., Magnier, E. A., Metcalfe, N., et al. 2016, arXiv e-prints, arXiv:1612.05560.  https://arxiv.org/abs/1612.05560
  * Dekany et al. (2020) Dekany, R., Smith, R. M., Riddle, R., et al. 2020, PASP, 132, 038001, doi: 10.1088/1538-3873/ab4ca2
  * Do et al. (2018) Do, A., Tucker, M. A., & Tonry, J. 2018, ApJ, 855, L10, doi: 10.3847/2041-8213/aaae67
  * Duev et al. (2016) Duev, D. A., Pogrebenko, S. V., Cimò, G., et al. 2016, A&A, 593, A34, doi: 10.1051/0004-6361/201628869
  * Duev et al. (2019) Duev, D. A., Mahabal, A., Masci, F. J., et al. 2019, MNRAS, 489, 3582, doi: 10.1093/mnras/stz2357
  * Duev (2020) Duev, D. A. e. a. 2020, Minor Planet Electronic Circulars, 2020-U170
  * Engelhardt et al. (2017) Engelhardt, T., Jedicke, R., Vereš, P., et al. 2017, AJ, 153, 133, doi: 10.3847/1538-3881/aa5c8a
  * Fitzsimmons et al. (2019) Fitzsimmons, A., Hainaut, O., Meech, K. J., et al. 2019, ApJ, 885, L9, doi: 10.3847/2041-8213/ab49fc
  * Géron (2019) Géron, A. 2019, Hands-on machine learning with Scikit-Learn and TensorFlow, second edition. edn. (O’Reilly Media, Inc.)
  * Giorgini et al. (1996) Giorgini, J. D., Yeomans, D. K., Chamberlin, A. B., et al. 1996, in AAS/Division for Planetary Sciences Meeting Abstracts, Vol. 28, AAS/Division for Planetary Sciences Meeting Abstracts #28, 25.04
  * Graham et al. (2019) Graham, M. J., Kulkarni, S. R., Bellm, E. C., et al. 2019, PASP, 131, 078001, doi: 10.1088/1538-3873/ab006c
  * Guzik et al. (2020) Guzik, P., Drahus, M., Rusek, K., et al. 2020, Nature Astronomy, 4, 53, doi: 10.1038/s41550-019-0931-8
  * Harris et al. (2020) Harris, C. R., Jarrod Millman, K., van der Walt, S. J., et al. 2020, arXiv e-prints, arXiv:2006.10256.  https://arxiv.org/abs/2006.10256
  * He et al. (2015) He, K., Zhang, X., Ren, S., & Sun, J. 2015, arXiv e-prints, arXiv:1512.03385.  https://arxiv.org/abs/1512.03385
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Ip et al. (2020) Ip, W. H., Bolin, B. T., Masci, F. J., et al. 2020, arXiv e-prints, arXiv:2009.04125.  https://arxiv.org/abs/2009.04125
  * Ivezić et al. (2008) Ivezić, Z., Tyson, J. A., Acosta, E., et al. 2008.  https://arxiv.org/abs/0805.2366v4
  * Jedicke et al. (2016) Jedicke, R., Bolin, B., Granvik, M., & Beshore, E. 2016, Icarus, 266, 173, doi: 10.1016/j.icarus.2015.10.021
  * Jensen-Clem et al. (2018) Jensen-Clem, R., Duev, D. A., Riddle, R., et al. 2018, AJ, 155, 32, doi: 10.3847/1538-3881/aa9be6
  * Kasliwal et al. (2019) Kasliwal, M. M., Cannella, C., Bagdasaryan, A., et al. 2019, Publications of the Astronomical Society of the Pacific, 131, 038003, doi: 10.1088/1538-3873/aafbc2
  * Kelley et al. (2019) Kelley, M. S. P., Bodewits, D., Ye, Q., et al. 2019, in Astronomical Society of the Pacific Conference Series, Vol. 523, Astronomical Data Analysis Software and Systems XXVII, ed. P. J. Teuben, M. W. Pound, B. A. Thomas, & E. M. Warner, 471
  * Kingma & Ba (2014) Kingma, D. P., & Ba, J. 2014, arXiv e-prints, arXiv:1412.6980.  https://arxiv.org/abs/1412.6980
  * Masci et al. (2019a) Masci, F. J., Laher, R. R., Rusholme, B., et al. 2019a, PASP, 131, 018003, doi: 10.1088/1538-3873/aae8ac
  * Masci et al. (2019b) —. 2019b, PASP, 131, 018003, doi: 10.1088/1538-3873/aae8ac
  * McCulloch & Pitts (1943) McCulloch, W. S., & Pitts, W. 1943, The bulletin of mathematical biophysics, 5, 115, doi: 10.1007/BF02478259
  * Mommert et al. (2019) Mommert, M., Kelley, M., de Val-Borro, M., et al. 2019, The Journal of Open Source Software, 4, 1426, doi: 10.21105/joss.01426
  * pandas development team (2020) pandas development team, T. 2020, pandas-dev/pandas: Pandas, latest, Zenodo, doi: 10.5281/zenodo.3509134
  * Schleicher et al. (1998) Schleicher, D. G., Millis, R. L., & Birch, P. V. 1998, 132, 397, doi: 10.1006/icar.1997.5902
  * Tan & Le (2019) Tan, M., & Le, Q. V. 2019, arXiv e-prints, arXiv:1905.11946.  https://arxiv.org/abs/1905.11946
  * Tan et al. (2019) Tan, M., Pang, R., & Le, Q. V. 2019, arXiv e-prints, arXiv:1911.09070.  https://arxiv.org/abs/1911.09070
  * Tonry et al. (2018) Tonry, J. L., Denneau, L., Heinze, A. N., et al. 2018, PASP, 130, 064505, doi: 10.1088/1538-3873/aabadf
  * van der Walt et al. (2019) van der Walt, S. J., Crellin-Quick, A., & Bloom, J. S. 2019, Journal of Open Source Software, 4, doi: 10.21105/joss.01247
  * Ye et al. (2020) Ye, Q., Masci, F. J., Ip, W.-H., et al. 2020, AJ, 159, 70, doi: 10.3847/1538-3881/ab629c
  * Zackay et al. (2016) Zackay, B., Ofek, E. O., & Gal-Yam, A. 2016, ApJ, 830, 27, doi: 10.3847/0004-637X/830/1/27


# Quantum Computing for Financial Mathematics

Antoine Jacquier  Department of Mathematics, Imperial College London, and the
Alan Turing Institute<EMAIL_ADDRESS>,  Oleksiy Kondratyev  Abu
Dhabi Investment Authority (ADIA), and Department of Mathematics, Imperial
College London<EMAIL_ADDRESS>,  Gordon Lee<EMAIL_ADDRESS>and
Mugad Oumgari  University College London and Lloyds Banking
<EMAIL_ADDRESS>
###### Abstract.

Quantum computing has recently appeared in the headlines of many scientific
and popular publications. In the context of quantitative finance, we provide
here an overview of its potential.

###### Key words and phrases:

quantum computing, quantitative finance

###### 2010 Mathematics Subject Classification:

68Q12, 68T07, 65D15

AJ is supported by the EPSRC grants EP/W032643/1 and EP/T032146/1.

Financial mathematics as a standalone discipline enjoyed many periods of rapid
development followed by periods of relative calm. As a synthetic discipline
that exists at the cross-section of applied mathematics, financial theory,
computer science, prudential regulation, …, it benefits from discoveries and
breakthroughs in all these fields.

The birth of financial mathematics is often attributed to the doctoral thesis
of Louis Bachelier, “The Theory of Speculation”, defended in 1900 (and written
under the supervision of Henri Poincaré) and which applied a stochastic
process (later called a Brownian motion) to the modelling of financial assets
for the first time. Since then, financial mathematics has been influenced by
stochastic calculus (Itô’s lemma, Girsanov’s theorem), control theory (Kalman
filter), statistics (Kolmogorov-Smirnov test), but also spurred by progresses
in microprocessors, financial deregulation, ultrafast communication, object-
oriented programming, to name just a few.

In this regard, quantum computing opens a new chapter for financial
mathematics as it promises enormous computing power at very low cost. To
understand why this is the case, one must appreciate the importance of
computation in general. At the end of the day, all financial problems
(pricing, risk management, credit scoring, discovery of trading signals, data
encryption, portfolio optimisation) are of computational nature, and the task
of financial mathematics is to provide the most efficient and convenient tools
to perform this computation.

What is computation? A computation can be defined as a transformation of one
memory state into another, or a function that transforms information. In the
case of classical digital computing, the fundamental memory unit is a binary
digit (bit) of information. Functions that operate on bits of information are
called logic gates, namely Boolean functions that can be combined into
circuits capable of performing additions, multiplications, and more complex
operations. But is the Boolean logic the only or even the most general way to
realise digital computation? The answer is clearly no. Classical computing is
just a special case of a more general computational framework now called
_quantum computing_. A classical bit is a two-state system that can exist in
either of two discrete deterministic states, traditionally denoted ‘0’ and
‘1’. All classical bits are independent – flipping the state of a given bit
does not affect the states of other bits. These two features of classical
computing can be generalised by allowing the bit to exist in a superposition
of the two states ‘0’ and ‘1’, and by allowing the states of different bits to
be entangled (a certain form of correlation). It is also clear how quantum
computing obtained its name – superposition and entanglement are the key
characteristics of the states of quantum systems, and it is tempting to
perform computation through the controlled evolution of a quantum system,
i.e., by running a quantum computer.

Superposition and entanglement are two features responsible for the
extraordinary power of quantum computing. They allow for more general
computation: more general definition of the memory state in comparison with
classical digital computing and a wider range of possible transformations of
such memory states. The fundamental memory unit in quantum computing is a
quantum binary digit (qubit). Mathematically, the state of a qubit is a unit
vector in the two-dimensional complex vector space. Quantum logic gates are
represented by the norm-preserving unitary operators (unitary matrices) acting
on qubit states. Once the computation is complete (the initial system state
has been transformed by the quantum circuit – a sequence of quantum logic
gates), the qubit states can be measured (projected onto the basis states).
Qubits in their basis states correspond to classical bits (all superpositions
have been collapsed). The rest of the computational protocol can be done
classically after the read-out of the bitstring, which is the output of the
quantum computer.

A natural question arises: what is the reason for this superior mode of
computation not being used until very recently? The answer is that although
quantum mechanics was formulated almost a century ago (Dirac’s seminal work
”The Principles of Quantum Mechanics” was published in 1930), the realisation
of the rules of quantum mechanics in the computational protocol performed on
classical digital computers requires an enormous amount of memory. Exponential
gains in computing power are offset by exponential memory requirements.

In order to perform quantum computations efficiently, we need to use actual
quantum mechanical systems, with their ability to encode information in their
states. To illustrate this point, the state of a quantum system consisting of
$n$ entangled qubits can be described by specifying $2^{n}$ probability
amplitudes, a huge amount of information even for very small systems ($n\sim
100$) and it would be impossible to store this information in classical
memory. It took decades of technological progress before Quantum Processing
Units (QPUs)––devices that control quantum mechanical systems performing
computations––became feasible.

The current state-of-the-art QPUs have several hundred qubits and the qubit
fidelity is still insufficient for the fault-tolerant computation. However,
these systems are already large enough and qubit fidelity high enough for such
quantum computers to be useful. Among many possible quantum computing
technologies, two qubit types stand out as most developed (and promising):

  * •

Qubits made of superconducting circuits (coherence time: $\sim 10^{3}$µs) [20]

one-qubit gate |  | two-qubit gate |
---|---|---|---
Gate time: |  $\sim 10^{-2}$µs | Gate time: |  $\sim 10^{-2}$-$10^{-1}$µs
Fidelity: | 99.99% | Fidelity: | 99.9%
  * •

Qubits made of trapped ions (coherence time: $>10^{8}$µs) [3]

one-qubit gate |  | two-qubit gate |
---|---|---|---
Gate time: |  $\sim 1$-$10$µs | Gate time: |  $\sim 10$µs
Fidelity: | 99.9999% | Fidelity: | 99.9%

These qubit characteristics are indicative of the fact that we approaching the
threshold beyond which various error correction algorithms become feasible and
we may finally enter the era of fault-tolerant quantum computing.

Whilst quantum supremacy (a term coined by Preskill in 2012) was demonstrated
by Google in 2019 on a specially designed problem, recent years have witnessed
clear signs of quantum advantage—first productive applications of quantum
computers to real-world problems hard for classical computers.

That said, given the scarcity and cost of full-fledged quantum computers,
emulators play a pivotal role in the current quantum ecosystem. These
emulators simulate quantum operations on classical hardware, allowing
researchers and developers to design, test, and refine quantum algorithms
without requiring direct access to a quantum machine, moving forward the state
of the art in its research but by-passing their current limited availability.
Graphics Processing Units (GPUs), given their parallel processing
capabilities, have emerged as the go-to hardware for emulating quantum
systems. Their architecture is well-suited for handling the matrix operations
fundamental to quantum mechanics. Tech giants in recent years have been
creating frameworks for quantum computing in public: IBM’s Qiskit development
kit allows any python user to implement and test quantum algorithms. Google
provides the Cirq framework, enabling developers to create, edit, and invoke
quantum circuits on real and simulated quantum devices. Microsoft’s Quantum
Development Kit includes the Q$\sharp$ language, which can be used to write
quantum algorithms that are run on classical simulators. Xanadu’s pennylane
software framework is also specifically designed to implement quantum machine
learning tools.

Several areas deserve a particular attention as the most relevant financial
mathematics problems.

Optimisation
Digital quantum computing offers the possibility of solving NP-hard
combinatorial optimisation problems using variational methods such as the
Variational Quantum Eigensolver [14, 17] and the Quantum Approximate
Optimisation Algorithm [6]. Both algorithms can be used to solve a wide range
of finance-related optimisation problems [11]. Moreover, variational
algorithms are noise-resistant and, therefore, suitable [4, 21] for running on
the current generation of Noisy Intermediate-Scale Quantum (NISQ) computers
[2, 18]. Additionally, classically hard optimisation problems naturally lend
themselves to be addressed by analog quantum computers that realise the
principles of Adiabatic Quantum Computing. The flagship financial use case is
discrete portfolio optimisation where we can see the first experimental
evidences of a quantum speedup [22].

Quantum Machine Learning
It is a combination of quantum computing and AI that will likely generate the
most exciting opportunities, including a whole range of possible applications
in finance. We have already witnessed the first promising results achieved
with parameterised quantum circuits trained as either generative models (such
as Quantum Circuit Born Machine [12], which can be used as a synthetic data
generator) or discriminative models (such as Quantum Neural Networks [7] that
can be trained as classifiers). The possible use cases include market
generators, data anonymisers, credit scoring, and the generation of trading
signals. The Quantum Generative Adversarial Networks is another type of
generative QML model with significant potential [1, 5, 10, 16, 19]. Similar to
classical GANs, a quantum GAN is composed of a generator and of a
discriminator with the ability to distinguish quantum states. Since each
quantum state encodes a probability distribution, the quantum GAN
discriminator can be used to verify whether the datasets in question were
drawn from the same probability distribution with direct application to the
time series analysis, detection of structural breaks, and monitoring of alpha
decay.

PDE Solvers
Harrow, Hassidim and Lloyd [9] devised a quantum algorithm to solve linear
systems, beating classical computation times. Linear systems are ubiquitous in
applications, and many aspects of mathematical finance rely on being able to
solve such (low- or high-dimensional) systems. A particularly important
application is solving Partial Differential Equations (PDEs). A quantum
algorithm for linear PDEs can be used to efficiently price European and Asian
options in the Black-Scholes framework [8].

Quantum Monte Carlo
Montanaro [15] described a quantum algorithm which can accelerate Monte Carlo
methods in a very general setting. The algorithm estimates the expected output
value of an arbitrary randomised or quantum subroutine with bounded variance,
achieving a near-quadratic speedup over the best possible classical algorithm.
For the recent advances in quantitative finance applications we refer
interested readers to [13].

Quantum Semidefinite Programming
The key idea behind Quantum Semidefinite Programming (QSDP) is based on the
observation that a normalised positive semidefinite matrix can be naturally
represented as a quantum state. Operations on quantum states can sometimes be
computationally cheaper to perform on a quantum computer than classical matrix
operations. This idea prompted the development of quantum algorithms for
semidefinite programming. In finance, QSDP can be used for the maximum risk
analysis and robust portfolio construction [11].

As a conclusion, after decades of theoretical results and promises, quantum
computing is now progressively becoming a reality. While full-scale quantum
computers are not here yet to replace their classical counterparts, they are
nevertheless already useful both to speed up specific procedures in classical
algorithms (giving birth to the _hybrid classical-quantum era_) and to provide
new angles and new ways of thinking about old problems (so-called _quantum-
inspired algorithms_). Rather than giving way to quantum scepticism, we would
instead quote David Deustch: ‘Quantum computation is [$\cdots$] nothing less
than a distinctly new way of harnessing nature’ [_The Fabric of Reality_
(1997), Chapter 9], and embrace it as a new tool, that will help us apprehend
more accurately the numerous problems faced in Quantitative Finance.

## References

  * [1] A. Assouel, A. Jacquier, and A. Kondratyev, A quantum generative adversarial network for distributions, Quantum Machine Intelligence, 4 (2022), p. 28.
  * [2] K. Bharti, A. Cervera-Lierta, T. H. Kyaw, T. Haug, S. Alperin-Lea, A. Anand, M. Degroote, H. Heimonen, J. S. Kottmann, T. Menke, W.-K. Mok, S. Sim, L.-C. Kwek, and A. Aspuru-Guzik, Noisy intermediate-scale quantum (NISQ) algorithms, arXiv:2101.08448, (2021).
  * [3] C. Bruzewicz, J. Chiaverini, R. McConnell, and J. Sage, Trapped-ion quantum computing: progress and challenges, Applied Physics Reviews, 6 (2019).
  * [4] M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. McClean, K. Mitarai, X. Yuan, L. Cincio, and P. Coles, Variational quantum algorithms, Nature Reviews Physics, 3 (2021), pp. 625–644.
  * [5] S. Chakrabarti, H. Yiming, T. Li, S. Feizi, and X. Wu, Quantum Wasserstein generative adversarial networks, Advances in Neural Information Processing Systems, 32 (2019).
  * [6] E. Farhi, J. Goldstone, and S. Gutmann, A quantum approximate optimization algorithm, arXiv:1411.4028, (2014).
  * [7] E. Farhi and H. Neven, Classification with quantum neural networks on near term processors, arXiv:1802.06002, (2018).
  * [8] F. Fontanela, A. Jacquier, and M. Oumgari, A quantum algorithm for linear PDEs arising in finance, SIAM Journal on Financial Mathematics, 12 (2021), pp. 98–114.
  * [9] A. W. Harrow, A. Hassidim, and S. Lloyd, Quantum algorithm for linear systems of equations, Physical review letters, 103 (2009), p. 150502.
  * [10] Y. Huang, H. Lei, X. Li, and G. Yang, Quantum maximum mean discrepancy GAN, Neurocomputing, 454 (2021), pp. 88–100.
  * [11] A. Jacquier and O. Kondratyev, Quantum Machine Learning and Optimisation in Finance: On the Road to Quantum Advantage, Packt Publishing Ltd, 2022.
  * [12] A. Kondratyev, Non-differentiable learning of quantum circuit Born machine with genetic algorithm, Wilmott, 2021 (2021), pp. 50–61.
  * [13] Y. Li and A. Neufeld, Quantum Monte Carlo algorithm for solving Black-Scholes PDEs for high-dimensional option pricing in finance and its proof of overcoming the curse of dimensionality, arXiv:2301.09241, (2023).
  * [14] J. R. McClean, J. Romero, R. Babbush, and A. Aspuru-Guzik, The theory of variational hybrid quantum-classical algorithms, New Journal of Physics, 18 (2016), p. 023023.
  * [15] A. Montanaro, Quantum speedup of Monte Carlo methods, Proceedings of the Royal Society A, 471 (2015).
  * [16] M. Y. Niu, A. Zlokapa, M. Broughton, S. Boixo, M. Mohseni, V. Smelyanskyi, and H. Neven, Entangling quantum generative adversarial networks, Physical Review Letters, 128 (2022), p. 220505.
  * [17] A. Peruzzo, J. McClean, P. Shadbolt, M.-H. Yung, X.-Q. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’brien, A variational eigenvalue solver on a photonic quantum processor, Nature communications, 5 (2014).
  * [18] J. Preskill, Quantum computing in the NISQ era and beyond, Quantum, 2 (2018).
  * [19] H. Situ, Z. He, Y. Wang, L. Li, and S. Zheng, Quantum GAN for generating discrete distribution, Information Sciences, 538 (2020), pp. 193–208.
  * [20] A. Somoroff, Q. Ficheux, R. Mencia, H. Xiong, R. Kuzmin, and V. Manucharyan, Millisecond coherence in a superconducting qubit, Physical Review Letters, 130 (2023).
  * [21] D. Stilck França and R. Garcia-Patron, Limitations of optimization algorithms on noisy quantum devices, Nature Physics, 17 (2021), pp. 1221–1227.
  * [22] D. Venturelli and A. Kondratyev, Reverse quantum annealing approach to portfolio optimization problems, Quantum Machine Intelligence, 1 (2019), pp. 17–30.


# Deformable Convolutions and LSTM-based Flexible Event Frame Fusion Network
for Motion Deblurring

Dan Yang  Huawei Technologies Oy (Finland) Co. Ltd
dan.yang1<EMAIL_ADDRESS>
Mehmet Yamac  Huawei Technologies Oy (Finland) Co. Ltd
dan.yang1<EMAIL_ADDRESS>

###### Abstract

Event cameras differ from conventional RGB cameras in that they produce
asynchronous data sequences. While RGB cameras capture every frame at a fixed
rate, event cameras only capture changes in the scene, resulting in sparse and
asynchronous data output. Despite the fact that event data carries useful
information that can be utilized in motion deblurring of RGB cameras,
integrating event and image information remains a challenge. Recent state-of-
the-art CNN-based deblurring solutions produce multiple 2-D event frames based
on the accumulation of event data over a time period. In most of these
techniques, however, the number of event frames is fixed and predefined, which
reduces temporal resolution drastically, particularly for scenarios when fast-
moving objects are present or when longer exposure times are required. It is
also important to note that recent modern cameras (e.g., cameras in mobile
phones) dynamically set the exposure time of the image, which presents an
additional problem for networks developed for a fixed number of event frames.
A Long Short-Term Memory (LSTM)-based event feature extraction module has been
developed for addressing these challenges, which enables us to use a
dynamically varying number of event frames. Using these modules, we
constructed a state-of-the-art deblurring network, Deformable Convolutions and
LSTM-based Flexible Event Frame Fusion Network (DLEFNet). It is particularly
useful for scenarios in which exposure times vary depending on factors such as
lighting conditions or the presence of fast-moving objects in the scene. It
has been demonstrated through evaluation results that the proposed method can
outperform the existing state-of-the-art networks for deblurring task in
synthetic and real-world data sets.

##  1 Introduction

Typically, blur is a common distortion that can be attributed to a variety of
factors, including motion, camera shake, depth variations in the scene, and
others. In earlier approaches, blurring kernels were applied globally to sharp
images or applied locally to sharp images [13]. It is, however, a challenging
ill-posed problem to estimate both blur kernels and sharp images
simultaneously, which may require iterative computation. With the emergence of
neural networks, most recent approaches based on convolutional neural networks
(CNNs) have demonstrated significant improvements in reconstruction accuracy
and computation efficiency [19, 33, 42, 35, 3]. Typically, they do not require
kernel estimation and train directly to map blurry images to sharp images
using data sets that are more realistic, such as the GoPro [19] or REDS [18]
datasets. Although single-image deblurring networks have made significant
advances [35, 3, 1, 41], their performance is still inadequate in challenging
situations, particularly when dealing with fast-moving objects.

This study aims to improve the quality of blurry images captured by standard
cameras by leveraging the capabilities of an event camera. The bio-inspired
sensors used in event cameras are capable of detecting changes in the
brightness of pixels at the microsecond level, thus encoding spatiotemporal
information of rapidly changing scenes at the pixel level [15, 27, 29].
Deblurring tasks, especially for images of dynamic scenes with moving objects,
may benefit greatly from such information [21, 9, 39, 32].

A number of advantages are offered by event cameras over traditional cameras,
including a higher dynamic range (exceeding 120dB, as opposed to 60dB for
traditional cameras), higher temporal resolution, and lower power consumption
[5]. Nevertheless, event cameras differ from conventional RGB cameras in their
asynchronous output of data sequences, which makes applying conventional
computer vision algorithms to event data difficult. As compared to traditional
cameras, event cameras record only changes in the scene, leading to a sparse
and asynchronous output of data. While the use of event data for motion
deblurring of RGB cameras is valuable, the integration of event and image data
remains a challenge. Based on the assumption that blurry images can be
represented as integrals of sharp latent images, [21] developed a model for
deblurring intensity images by utilizing a double integral technique. Such
model-based algorithms are iterative and computationally demanding, which
makes them incompatible with portable devices such as smartphones. The
algorithms are also limited to grayscale images, so adapting them to RGB
images remains challenging.

Researchers have recently focused on integrating event data over a time period
in order to create multi-frame event data that is more compatible with
computer vision algorithms developed for RGB frame processing. This approach
enables the use of multi-frame event data alongside RGB frames for tasks such
as deblurring [39, 32]. The current literature, however, fixes the number of
event frames by assuming a fixed exposure time for all the images, leading to
a reduced temporal resolution when dealing with fast-moving objects or when a
longer exposure time is necessary. As an example, in mobile cameras, the
exposure time varies dynamically, which poses a particular challenge.

The purpose of this study is to develop a neural network structure for event-
data-assisted deblurring that can accommodate dynamic changes in the number of
event frames being constructed. This network is suitable for two realistic
scenarios or a combination of them: (i) Modern cameras, including mobile
cameras, are capable of setting the exposure time automatically based on
factors such as lighting conditions, ISO, aperture, and shutter speed when
capturing images. (ii) If one desires to use a fixed exposure time for an RGB
camera, integrating event data using uniformly divided time interval bins to
produce event frames is not optimal. In order to achieve better results,
smaller bins (i.e., having more event frames) can be used when more event
sensors are triggered, such as when cameras are moving faster or when objects
are moving faster.

First, we have developed a feature extraction module based on Long Short-Term
Memory (LSTM) that allows us to use a dynamic number of event frames during
feature extraction. Then, we have used deformable CNNs in the LSTM module in
order to increase the functionality of feature extraction. Afterward, the
extracted features are combined with the encoded features of RGB frames in
multiple scales to construct a state-of-the-art deblurring network, DLEFNet.
In addition to its ability to handle fast-moving objects and varying exposure
times, our model proves particularly valuable in real-world scenarios where
the scene properties is unknown in advance. The proposed solution has achieved
new state-of-the-art levels of image reconstruction accuracy for both the
GoPro dataset [19] and the recently released real event data deblurring
dataset, REBlur [32].

##  2 Related Works

An event camera detects changes in intensity and records the pixel location,
change time, and polarity of change. As a result, event cameras obtain sparse
and asynchronous information with high temporal resolution, which is
significantly different from conventional standard image representation. This
makes it challenging to use event data with conventional computer vision
algorithms.

One approach is to consider a standard camera coupled with an event camera. In
this approach, the blurry image intensity can be regarded as the integral over
sharp latent images changing with time, and each small change between any so-
called sharp images can be calculated over the sum of the event stream. This
assumption led to the proposal of a model for deblurring intensity images
using a double integral technique [21]. After that, a deep unfolding neural
network has been proposed to model deblurring as a Maximum-a-Posteriori (MAP)
problem and solve it [9].

Recent works have shown that event streams can be converted to event frames by
simple accumulation for various computer vision tasks such as image and video
deblurring [39, 32], depth estimation [12], optical flow estimation [16], and
feature tracking [7]. Although neural network-based solutions like MADANet and
EFNet [39, 32] that feed the network with blurry images and event frames
obtained through simple uniform quantization with a fixed number of bins
achieve state-of-the-art deblurring results, there is still significant room
for improvement, particularly in event feature extraction and/or fusion steps.
This is because these methods consider a fixed and generally small number of
uniformly split bins (e.g., 5, 6, or 7), which fails to utilize the full
temporal information that event data provides.

In this work, we propose a novel event-data-assisted deblurring neural network
structure that can adapt to the changing number of event frames between
captures. This approach offers two significant benefits: (a) it makes the
algorithm more suitable for modern cameras whose exposure time dynamically
changes, and (b) it allows for smaller interval bins (more event frames) than
the literature, which enables the network to utilize more temporal information
provided by event data.

We have developed a novel approach for deblurring networks that leverages a
feature extraction module based on Long Short-Term Memory (LSTM) capable of
utilizing a dynamic number of event frames during feature extraction.
Furthermore, we have incorporated deformable convolutional neural networks
(CNNs) within the LSTM module to enhance the functionality of feature
extraction. The extracted features are then combined with the encoded features
of RGB frames across multiple scales to create a state-of-the-art deblurring
network. Our model demonstrates superior performance in scenarios where the
exposure time is unknown in advance, making it particularly valuable.
Additionally, the model is capable of handling fast-moving objects and varying
exposure times.

In a recent paper [38], convolutional LSTMs and deformable convolutions are
utilized as well, but separately. The former is used to extract event
features, whereas the latter is used to combine image features with event
features. Their aim is to create local motion-aware convolution. However, they
still consider a fixed number of event frames (i.e., N=5). The study [11]
might be more relevant to our target task. The study [11] attempted to handle
dynamically changing video deblurring exposure times. However, they still
assumed a fixed shutter period, divided into an unknown exposure time and an
unknown readout time. Essentially, they considered a predefined number of
frames, $N=m+n$, and used an event feature selective module to select features
from the first $m$ frames when $m$ was unknown. In contrast, our study focuses
on single image deblurring with an unknown exposure time and therefore an
unknown number of event frames.

##  3 Proposed Solution

###  3.1 Event Data Pre-processing

In contrast to traditional cameras, newly emerging event cameras do not have a
predetermined exposure time. In contrast, they detect changes in brightness
asynchronously at the microsecond level within a scene. A sequence of events,
denoted by $\left(x,y,t,\delta\right)$, is output by event cameras instead of
capturing the intensity at pixel $\left(x,y\right)$ at time $t$. The time of
occurrence of the event is represented by $t$, and the polarity of the event
is represented by $\delta$:

$\delta=\begin{cases}+1,&g\geq thr\\\ -1&g\leq-thr\\\ 0&\text{else}\\\
\end{cases}$

where $thr$ is the threshold determined by the the user, $I_{x,y}$ is the
intensity level at pixel $\left(x,y\right)$ and the proportional change
calculated logarithmically, i.e.,
$g=\log\left(\frac{I_{x,y}\left(t\right)}{I_{x,y}\left(t_{ref}\right)}\right)$,
when $t_{ref}$ is the previous event occurrence time.

For the purpose of processing these events, instead of dividing the event
stream into several chunks during the standard and fixed exposure time, $T$,
as is done in literature [39, 32], we consider that $T$ is neither fixed nor
known. We accumulate event data over very short intervals of time, $\Delta T$
(e.g., 1 or 2 milliseconds). A ternary 2-D frame is then constructed from the
accumulated data, i.e.

$e_{x,y}^{i}=Q\left(\int_{\left(i-1\right)\Delta T}^{i\Delta
T}\varepsilon_{x,y}\left(t\right)dt\right)~{}~{}\text{for}~{}~{}i=1,2,..$ (1)

where $Q(h)=\text{sgn}(h)$ and $\varepsilon_{x,y}\left(t\right)$ corresponds
to an individual event occurring at time $t$ and at pixel location $(x,y)$.
Unlike similar event representation schemes [12, 16, 7], our event frames have
ternary data, $e_{x,y}^{i}\in\left\\{1,0,-1\right\\}$. As the exposure time,
$T$, is unknown in advance, the number of event frames varies from image to
image, i.e., $n=\frac{T}{\Delta T}$.

###  3.2 Deformable Convolutional LSTM for Event Feature Extraction

Let $\mathbf{B}\in\mathbb{R}^{3\times H\times W}$ be the blurry RGB image. The
Dynamic Event Frame Feature Extraction (DEFFE) module uses the dynamically
changing number of event frames and the blurry image in order to get $C$
number of event feature maps using Bidrectional Deformable Convolution Long
Short-Short Term Memory (BiDefConvLSTM) cells in recurrent manner.

The Deformable Convolution LSTM (DefConvLSTM) cell used in this module is
formulated as follows:

$\displaystyle i_{t}=$
$\displaystyle\sigma\left(W_{Ei}\otimes\mathbf{e^{t}}+W_{Hi}\otimes\mathbf{H_{t-1}}+W_{Bi}\otimes\mathbf{B}+b_{i}\right),$
(2) $\displaystyle f_{t}=$
$\displaystyle\sigma\left(W_{Ef}\otimes\mathbf{e^{t}}+W_{Hf}\otimes\mathbf{H_{t-1}}+W_{Bf}\otimes\mathbf{B}+b_{f}\right),$
(3) $\displaystyle o_{t}=$
$\displaystyle\sigma\left(W_{Eo}\otimes\mathbf{e^{t}}+W_{Ho}\otimes\mathbf{H_{t-1}}+W_{Bo}\otimes\mathbf{B}+b_{o}\right),$
(4) $\displaystyle c_{t}=$
$\displaystyle\sigma\left(W_{Ec}\otimes\mathbf{e^{t}}+W_{Hc}\otimes\mathbf{H_{t-1}}+W_{Bc}\otimes\mathbf{B}+b_{c}\right),$
(5) $\displaystyle C_{t}=$ $\displaystyle f_{t}\odot tanh\left(c_{t}\right)$
(6) $\displaystyle H_{t}=$ $\displaystyle o_{t}\odot tanh\left(C_{t}\right)$
(7) $\displaystyle w_{t}=$ $\displaystyle ha\left(\textbf{H}_{t}\right)$ (8)
$\displaystyle\textbf{Fe}=$ $\displaystyle\mathbf{Fe}+w_{t}\mathbf{H}_{t},$
(9)

where $\otimes$ denotes the deformable convolutional operator, $\odot$ denotes
the Hadamard product, $\sigma$ denotes the Sigmoid operation, $i_{t}$,
$f_{t}$, $o_{t}$ and $c_{t}$ denotes input, forget, output, and cell gates,
$W_{E*},W_{H*},W_{B*}$ are deformable convolutional kernels, $b_{*}$ are bias
terms, $w_{t}$ is hidden layer attention weight,
$\mathbf{Fe}\in\mathbb{R}^{C\times H\times W}$ is $C$-channel extracted
feature map and $ha(.)$ is hidden layer attention operation. The hidden layer
attention operation is simply channel-wise squeeze operation followed by MLP
layers and a sigmoid, i.e., for a feature map
$\mathbf{X}\in\mathbb{R}^{C\times H\times W}$, it is defined as

$ha(\mathbf{X})=\sigma\left(MLP\left(GAP\left(\mathbf{X}\right)\right)\right)\in\mathbb{R}^{1\times
1\times 1}.$ (10)

The proposed LSTM module works on Bidrectional manner in a way that as forfard
process the above mentioned LSTM cells takes firt half of the event frames in
forward manner, i.e., $1,2,...,n/2$. On the other hand, as bacward process the
cells takes the second half in reverse order, i.e., $n,n-1,...,n/2$.

Figure 1: An example pair of forward and backward feature maps.

###  3.3 The overall network

DLEFNet consists of two main parts: the first is the DEFFE Module, which was
explained above, and the other is the main body deblurring module, denoted by
$\mathcal{M}$. The deblurring module is an adaptation of the Frequency Domain
Multiple Input Multiple Output Unet (FMIMO-UNet) structure and consists of an
encoder, $\mathcal{M_{E}}$, and a decoder, $\mathcal{M_{D}}$. Both the encoder
and decoder have a multilevel structure, with three levels.

Let $\mathcal{M_{E}}_{L}$ be the $L^{th}$ level encoder block, which takes as
input $\mathbf{F{in}}^{L}\in\mathbf{R}^{c_{i}\times h_{i}\times w_{i}}$ and
$\mathbf{Fe_{in}}^{L}\in\mathbf{R}^{c_{i}\times h_{i}\times w_{i}}$, and
produces as output $\mathbf{F_{out}}^{L}\in\mathbf{R}^{c_{o}\times h_{o}\times
w_{o}}$. Specifically,

$\mathbf{F_{out}^{L}}\leftarrow\mathcal{M_{E}}_{L}\left(\mathbf{F{in}}^{L},\mathbf{Fe_{in}}^{L}\right),$
(11)

where $\mathbf{F_{in}}^{L}$ is the input image feature map with $c_{i}$
channels, and $\mathbf{Fe_{in}}^{L}$ is the event feature map for the $L^{th}$
level. For the first level, the encoder module takes the 3-channel RGB blurry
image, $\mathbf{F_{in}}^{L}=\mathbf{B}$. A convolutional layer with a stride
of 2 is applied to $\mathbf{F_{out}^{L}}$ to obtain $\mathbf{F_{in}}^{L+1}$,
which doubles the channel dimension and reduces the height and width by a
scale factor of 2.

Each $\mathcal{M_{E}}_{L}$ first processes the image feature maps using
frequency domain residual blocks, as proposed in [17]. To fuse the event
feature maps at each level, the Event-Image Crossmodal Attention (EICA) fusion
module, borrowed from [32], is used. For the decoder, $\mathcal{M_{D}}$, the
features are processed by frequency domain residual blocks, and multi-scale
feature maps are fused using asymmetric feature fusion (AFF) modules, as done
in MIMO-Unet [3]. The event features from DEFFE moudule is also passed through
an $L$ level event encoder block, $\mathcal{M_{E}}^{E}$ before fusing them
with image feature maps. The overall structure of the network can be found in
Figure 2.

Figure 2: The pipeline of the proposed DLEFNet

##  4 Network Details and Training

In this study, we have designed our U-Net architecture with a symmetrical
structure. We set the number of output channels to 32 for the first level, and
then increased it to 64 and 128 in the following layers of the encoder. For
the decoder, the number of channels was set to 128, 64, and 32, respectively.
We used $3\times 3$ kernels for all the convolutional layers. The number of
event feature maps was set to 32, which were obtained by considering 16
forward and 16 backward directions. We employed 8 residual blocks at each
level. To compute the loss, we utilized a hybrid loss function comprising
$\mathcal{L}_{1}$ ($\ell_{1}$-norm metric) and a SSIM-based loss
$\mathcal{L}_{s}$, i.e., $\mathcal{L}=\mathcal{L}_{1}+0.5\mathcal{L}_{s}$. We
applied a cosine scheduler that decreased the learning rate from 0.001 to
1e-7. In MLP block given in Equation (10), 32 channel was map to 2, then a
ReLu activation function was used before last layer mapping 2 dimensional
scalar value defining the which hidden layers are more relevant.

##  5 Experimental Evaluation

Table 1: Deblurring results on GoPro dataset Method | PSNR | SSIM | Params
---|---|---|---
BHA [22] | 29.06 | 0.943 |  N/A
DeepDeblur [20] | 29.23 | 0.916 | 11.7M
SVDN [40] | 29.81 | 0.937 |  N/A
SRN [34] | 30.26 | 0.934 | 6.8M
DGN [14] | 30.49 | 0.938 | 11.32M
PSS-NSC [6] | 30.92 | 0.942 | 2.8M
MT-RNN [24] | 31.15 | 0.945 | 2.6M
DMPHN [44] | 31.20 | 0.945 | 21.7M
RADN [25] | 31.76 | 0.953 | N/A
LEBMD [10] | 31.79 | 0.949 |  N/A
PVDNet [28] | 31.98 | 0.928 | 23.4M
SAPHN [30] | 32.02 | 0.953 |  N/A
GSTA [31] | 32.10 | 0.960 |  N/A
MBRNN [23] | 32.16 | 0.953 | 5.42M
BANET [36] | 32.44 | 0.957 | 85.6M
MPRNET [43] | 32.66 | 0.959 | 20.1M
MIMO-UNet++ [4] | 32.68 | 0.959 | 16.1M
HINet [2] | 32.71 | 0.959 | 88.6M
Restormer [41] | 32.92 | 0.961 | 26.1
ERDN [8] | 32.99 | 0.935 | N/A
NAFNet [1] | 33.71 | 0.967 | 67.9
MADANET+ [39] | 33.84 | 0.964 | 16.9M
Vitoria et al. [37] | 34.33 | 0.944 | N/A
EFNet [32] | 35.46 | 0.972 | 8.5M
DLEFNet (ours) | 35.61 | 0.973 | 12M

###  5.1 GoPro Dataset

Figure 3: Visual comparison of the state-of-the-art algorithms on GoPro
dataset

We adhere to the recommended protocol for training and testing by averaging a
varying number of nearby frames (ranging from 7 to 13) to generate the blurry
image. In that manner, the synthetic events corresponding to the averaged
frames are accumulated into varying number of event frames ranging from 7 to
13. ESIM simulator [26] was used to generate the corresponding events. We use
a total of 2103 samples for training, consisting of blurry image, event frames
stack, and ground-truth frames, and 1111 samples for testing. Table 1 compares
the performance of DLEFNet with other state-of-the-art deblurring methods. BHA
[21], LEBMD [9], ERDN, Vitoria et al. [38], and EFNet [32] are event-aided
solutions, while DGN [14] provides depth-aware deblurring. MBRNN [23], GSTA
[31], and PVDNet [28] are video-based (multi-frame) methods. The proposed
solution significantly outperforms the SoTa deblurring techniques as can be
seen in the Table 1. Visual comparison of the algorithms are given in Figure
3.

###  5.2 REBlur Dataset

Figure 4: Visual comparison of the state-of-the-art algorithms on REBlur
dataset

The Real Event Blur (REBlur) dataset is a recently proposed collection of
event data and real blurry images [32]. REBlur encompasses 12 different types
of linear and nonlinear motions for 3 distinct moving patterns and for the
camera itself. It includes 36 sequences and 1469 pairs of blurry-sharp images
with associated events. Of these pairs, 486 are used for training, and 983 for
testing. As done in [32], we fine-tuned our network trained on the GoPro
dataset with the training portion of the REBlur dataset. The evaluation was
then conducted on the test set. The performance comparison of algorithms is
presented in Table 2. The results of the algorithms with a ”+” are from [32],
in which the blurry images and event frames produced in [32] are fed and
trained together. A visual comparison of EFNet and DLEFNet is provided in
Figure 4. Our proposed solution outperforms the state-of-the-art network EFNet
both quantitatively and visually.

Table 2: Deblurring results on REBlur dataset Method | PSNR | Params
---|---|---
SRN [34] | 35.10 | 10.25M
NAFNet [1] | 35.48 | 67.89M
Restormer [41] | 35.50 | 26.13M
HINet [2] | 35.58 | 88.67M
BHA+ [22] | 36.62 | 0.51M
SRN+ [34] | 36.87 | 10.43M
HINet+ [2] | 37.68 | 88.85M
EFNet [32] | 38.12 | 8.47M
DLEFNet (ours) | 38.40 | 12M

##  6 Conclusions

Event cameras differ from RGB cameras in that they produce asynchronous data
sequences by capturing only changes in the scene. This unique property makes
it challenging to utilize existing computer vision algorithms with event data.
Recent state-of-the-art solutions generate event frames by dividing the
exposure time, assuming it remains constant between frames, into a fixed
number of smaller time intervals. However, this approach is unrealistic as
modern cameras automatically adjust exposure time based on scene information
obtained from multiple sensors. Additionally, fixing the number of frames
results in large time interval bins to compensate for sparse event data,
causing reduced temporal resolution in scenarios with fast-moving objects or
longer exposure times. To address these issues, the Deformable Convolutions
and LSTM-based Flexible Event Frame Fusion Network (DLEFNet) has been
proposed, demonstrating superior performance compared to existing deblurring
solutions on both synthetic and real-world datasets such as GoPro and REBlur.

## References

  * [1] Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII. pp. 17–33. Springer (2022)
  * [2] Chen, L., Lu, X., Zhang, J., Chu, X., Chen, C.: Hinet: Half instance normalization network for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 182–192 (2021)
  * [3] Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4641–4650 (2021)
  * [4] Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. arXiv preprint arXiv:2108.05054 (2021)
  * [5] Gallego, G., Delbruck, T., Orchard, G.M., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A., Conradt, J., Daniilidis, K., Scaramuzza, D.: Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 1–1 (2020). https://doi.org/10.1109/TPAMI.2020.3008413
  * [6] Gao, H., Tao, X., Shen, X., Jia, J.: Dynamic scene deblurring with parameter selective sharing and nested skip connections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3848–3856 (2019)
  * [7] Gehrig, D., Rebecq, H., Gallego, G., Scaramuzza, D.: Asynchronous, photometric feature tracking using events and frames. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 750–765 (2018)
  * [8] Haoyu, C., Minggui, T., Boxin, S., YIzhou, W., Tiejun, H.: Learning to deblur and generate high frame rate video with an event camera. arXiv preprint arXiv:2003.00847 (2020)
  * [9] Jiang, Z., Zhang, Y., Zou, D., Ren, J., Lv, J., Liu, Y.: Learning event-based motion deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3320–3329 (2020)
  * [10] Jiang, Z., Zhang, Y., Zou, D., Ren, J., Lv, J., Liu, Y.: Learning event-based motion deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3320–3329 (2020)
  * [11] Kim, T., Lee, J., Wang, L., Yoon, K.J.: Event-guided deblurring of unknown exposure time videos. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII. pp. 519–538. Springer (2022)
  * [12] Kogler, J., Sulzbachner, C., Kubinger, W.: Bio-inspired stereo vision system with silicon retina imagers. In: International Conference on Computer Vision Systems. pp. 174–183. Springer (2009)
  * [13] Kundur, D., Hatzinakos, D.: Blind image deconvolution. IEEE signal processing magazine 13(3), 43–64 (1996)
  * [14] Li, L., Pan, J., Lai, W.S., Gao, C., Sang, N., Yang, M.H.: Dynamic scene deblurring by depth guided model. IEEE Transactions on Image Processing 29, 5273–5288 (2020)
  * [15] Lichtsteiner, P., Posch, C., Delbruck, T.: A 128 x 128 120db 30mw asynchronous vision sensor that responds to relative intensity change. In: 2006 IEEE International Solid State Circuits Conference-Digest of Technical Papers. pp. 2060–2069. IEEE (2006)
  * [16] Liu, M., Delbruck, T.: Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors (2018)
  * [17] Mao, X., Liu, Y., Shen, W., Li, Q., Wang, Y.: Deep residual fourier transformation for single image deblurring. arXiv preprint arXiv:2111.11745 (2021)
  * [18] Nah, S., Baik, S., Hong, S., Moon, G., Son, S., Timofte, R., Mu Lee, K.: Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 0–0 (2019)
  * [19] Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3883–3891 (2017)
  * [20] Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3883–3891 (2017)
  * [21] Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., Dai, Y.: Bringing a blurry frame alive at high frame-rate with an event camera. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6820–6829 (2019)
  * [22] Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., Dai, Y.: Bringing a blurry frame alive at high frame-rate with an event camera. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6820–6829 (2019)
  * [23] Park, D., Kang, D.U., Chun, S.Y.: Blur more to deblur better: Multi-blur2deblur for efficient video deblurring. arXiv preprint arXiv:2012.12507 (2020)
  * [24] Park, D., Kang, D.U., Kim, J., Chun, S.Y.: Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: European Conference on Computer Vision. pp. 327–343. Springer (2020)
  * [25] Purohit, K., Rajagopalan, A.: Region-adaptive dense network for efficient motion deblurring. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 11882–11889 (2020)
  * [26] Rebecq, H., Gehrig, D., Scaramuzza, D.: Esim: an open event camera simulator. In: Conference on Robot Learning. pp. 969–982. PMLR (2018)
  * [27] Son, B., Suh, Y., Kim, S., Jung, H., Kim, J.S., Shin, C., Park, K., Lee, K., Park, J., Woo, J., et al.: 4.1 a 640$\times$ 480 dynamic vision sensor with a 9$\mu$m pixel and 300meps address-event representation. In: 2017 IEEE International Solid-State Circuits Conference (ISSCC). pp. 66–67. IEEE (2017)
  * [28] Son, H., Lee, J., Lee, J., Cho, S., Lee, S.: Recurrent video deblurring with blur-invariant motion estimation and pixel volumes. ACM Transactions on Graphics (TOG) 40(5), 1–18 (2021)
  * [29] Suh, Y., Choi, S., Ito, M., Kim, J., Lee, Y., Seo, J., Jung, H., Yeo, D.H., Namgung, S., Bong, J., et al.: A 1280$\times$ 960 dynamic vision sensor with a 4.95-$\mu$m pixel pitch and motion artifact minimization. In: 2020 IEEE international symposium on circuits and systems (ISCAS). pp. 1–5. IEEE (2020)
  * [30] Suin, M., Purohit, K., Rajagopalan, A.: Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3606–3615 (2020)
  * [31] Suin, M., Rajagopalan, A.: Gated spatio-temporal attention-guided video deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7802–7811 (2021)
  * [32] Sun, L., Sakaridis, C., Liang, J., Jiang, Q., Yang, K., Sun, P., Ye, Y., Wang, K., Gool, L.V.: Event-based fusion for motion deblurring with cross-modal attention. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII. pp. 412–428. Springer (2022)
  * [33] Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8174–8182 (2018)
  * [34] Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8174–8182 (2018)
  * [35] Tsai, F.J., Peng, Y.T., Lin, Y.Y., Tsai, C.C., Lin, C.W.: Banet: Blur-aware attention networks for dynamic scene deblurring. arXiv preprint arXiv:2101.07518 (2021)
  * [36] Tsai, F.J., Peng, Y.T., Lin, Y.Y., Tsai, C.C., Lin, C.W.: Banet: Blur-aware attention networks for dynamic scene deblurring. arXiv preprint arXiv:2101.07518 (2021)
  * [37] Vitoria, P., Georgoulis, S., Tulyakov, S., Bochicchio, A., Erbach, J., Li, Y.: Event-based image deblurring with dynamic motion awareness. arXiv preprint arXiv:2208.11398 (2022)
  * [38] Vitoria, P., Georgoulis, S., Tulyakov, S., Bochicchio, A., Erbach, J., Li, Y.: Event-based image deblurring with dynamic motion awareness. In: Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part V. pp. 95–112. Springer (2023)
  * [39] Yang, D., Yamac, M.: Motion aware double attention network for dynamic scene deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1113–1123 (2022)
  * [40] Yuan, Y., Su, W., Ma, D.: Efficient dynamic scene deblurring using spatially variant deconvolution network with optical flow guided training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3555–3564 (2020)
  * [41] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5728–5739 (2022)
  * [42] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14821–14831 (2021)
  * [43] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14821–14831 (2021)
  * [44] Zhang, H., Dai, Y., Li, H., Koniusz, P.: Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5978–5986 (2019)


By induction define for every $j=1,...,k$ a sequence of integers
$(q(i,j))_{i=0}^{m_{j}}$ as follows. Set $q(i,1)=i$ for all $i=0,...,m_{1}$
and for every $j=1,...,k-1$ set
$q(i,j+1)=q(q_{\mathrm{Sh}^{*}}(i,k,\kappa,m_{j+1},r),j)$ for all
$i=0,...,m_{j+1}$. Notice that $m_{k}=m$. Set $q_{i}=q(i,k)$ for all
$i=0,...,m$.

To complete the proof we show that $n_{0}$ and $(q_{i})_{i=0}^{m}$ satisfy the
conclusion of the theorem. By the definition of these quantities, we have that
$0=q_{0}<q_{1}<...<q_{m}\leqslant n_{0}$. Let $N$ be an integer with
$N\geqslant n_{0}$, let $\Lambda$ be a finite alphabet with $k$ elements and
let $c$ be an $r$-coloring of $\Lambda^{N}\times\\{0,...,N-1\\}^{(\kappa)}$.
Write $\Lambda=\\{a_{1},...,a_{k}\\}$ and set $L_{j}=\\{a_{j},a_{j+1}\\}$ for
all $j=1,...,k-1$. Inductively we construct a decreasing sequence
$(X_{j})_{j=1}^{k}$ of $\kappa^{*}$combinatorial subspaces of
$\Lambda^{N}\times\\{0,...,N-1\\}^{(\kappa)}$ such that for every $j=1,...,k$
the following are satisfied.

  1. (i)

The subspace $X_{j}$ is $m_{j}$-dimensional and
$(q(i,j))_{i=0}^{m_{j}}$-compatible.

  2. (ii)

If $j>1$ then the coloring $c$ is $L_{j-1}^{*}$insensitive in $X_{j}$.

We set
$w_{1}(v_{0},...,v_{n_{0}-1})=(v_{0},...,v_{n_{0}-1})^{\smallfrown}\mathbf{x}$,
where $\mathbf{x}$ is a constant word over $\Lambda$ of length $N-n_{0}$. Let
$X_{1}$ be the $\kappa^{*}$combinatorial subspace generated by $w_{1}$. Assume
that for some $j\in\\{1,...,k-1\\}$ we have constructed the properly the
subspaces $X_{1},...,X_{j}$. We describe the construction of $X_{j+1}$. Let
$w_{j}(v_{0},...,v_{m_{j}-1})$ be the $m_{j}$-variable word generating
$X_{j}$. Define an $r$-coloring $\tilde{c}$ of
$\Lambda^{m_{j}}\times\\{0,...,m_{j}-1\\}^{(\kappa)}$ setting for every
$(\mathbf{a},F)$ in $\Lambda^{m_{j}}\times\\{0,...,m_{j}-1\\}^{(\kappa)}$

$\tilde{c}(\mathbf{a},F)=c(w_{j}(a_{0},...,a_{m_{j}-1}),\\{\ell^{w_{j}}_{i}:i\in
F\\}),$

where $\mathbf{a}=(a_{i})_{i=0}^{m_{j}-1}$. By Lemma 31, there exists an
$m_{j+1}$-dimensional $\kappa^{*}$combinatorial
$(q_{\mathrm{Sh}^{*}}(i,k,\kappa,m_{j+1},r))_{i=0}^{m_{j+1}}$-compatible
subspace $Y$ such that $\tilde{c}$ is $L_{j}^{*}$insensitive in $Y$. Set
$X_{j+1}=X_{j}[Y]$. Then $X_{j+1}$ is an $m_{j+1}$-dimensional further
$\kappa^{*}$subspace of $X_{j}$ and $c$ is $L_{j}^{*}$insensitive in
$X_{j+1}$. Moreover, by Remark 4, the fact that $X_{j}$ is
$(q(i,j))_{i=0}^{m_{j}}$-compatible and the definition of the numbers
$((q(i,j))_{i=0}^{m_{j}})_{j=1}^{k}$, we have that $X_{j+1}$ is
$(q(i,j+1))_{i=0}^{m_{j+1}}$-compatible. The inductive construction of
$(X_{j})_{j=1}^{k}$ is complete.

Set $X=X_{k}$. Notice that $m_{k}=m$. Thus, invoking the definition of the
sequence $(q_{i})_{i=0}^{m}$, we have that $X$ is an $m$-dimensional
$\kappa^{*}$combinatorial $(q_{i})_{i=0}^{m}$-compatible subspace of
$\Lambda^{N}\times\\{0,...,N-1\\}^{(\kappa)}$. Moreover, since $X$ is a
subspace of $X_{j}$ for every $j=2,...,k$, by condition (ii) of the inductive
construction of $(X_{j})_{j=1}^{k}$ and Remark 3, we have that c is strongly
∗insensitive in $X$. ∎

## Acknowledgment

The research was supported by the Hellenic Foundation for Research and
Innovation (H.F.R.I.) under the “2nd Call for H.F.R.I. Research Projects to
support Faculty Members & Researchers” (Project Number: HFRI-FM20-02717).

The research on this paper is partially supported by grants from NSERC(455916)
and CNRS(UMR7586).

## References

  * [1] R. L. Graham and B. L. Rothschild, Ramsey’s theorem for $n$-parameter sets, Trans. Amer. Math. Soc., 159 (1971), 257-292.
  * [2] R.L. Graham, B.L. Rothschild, and J. H. Spencer,  Ramsey theory. Second edition. Wiley-Interscience Series in Discrete Mathematics and Optimization. A Wiley-Interscience Publication. John Wiley and Sons, Inc., New York, 1990. xii+196 pp.
  * [3] A. H. Hales and R. I. Jewett, Regularity and positional games, Trans. Amer. Math. Soc., 106 (1963), 222–229.
  * [4] N. Hindman, Finite sums from sequences within cells of a partition of $\mathbb{N}$, J. Comb. Theory, Ser. A 17 (1974), 1–11.
  * [5] K. R. Milliken, Ramsey’s theorem with sums or unions, J. of Comb. Th., Series A, 18 (1975): 276–290.
  * [6] K. Milliken, A Ramsey theorem for trees, J. Comb. Theory Ser. A, 26 (1979), 215-237.
  * [7] S. Shelah, Primitive recursive bounds for van der Waerden numbers, J. Amer. Math. Soc. 1 (1988), 683–697.
  * [8] M. Sokić, Bounds on trees, Discrete Math. 311 (2011), no. 6, 398–407.
  * [9] A. D. Taylor, A canonical partition relation for finite subsets of $\omega$, J. of Comb. Th., Series A, 21 (1976): 137–146.
  * [10] S. Todorcevic, Introduction to Ramsey Spaces, Annals Math. Studies, No. 174, Princeton Univ. Press, 2010.
  * [11] S. Todorcevic and K. Tyros, A disjoint union theorem for trees, Adv. in Math. 285 (2015), 1487–1510.