# Unsupervised Traffic Scene Generation with Synthetic 3D Scene Graphs

Artem Savkin1,2
TUM, BMW  Rachid Ellouze1,2
TUM, BMW  Nassir Navab1,3
TUM, JHU  Federico Tombari1,4
TUM, Google 1TUM, Boltzmannstr. 3, 85748 Munich, Germany<EMAIL_ADDRESS><EMAIL_ADDRESS>AG, 80809 Munich, Germany1JHU, Baltimore, MD 21218,
United States4Google, 8002 Zurich, Switzerland

###### Abstract

Image synthesis driven by computer graphics achieved recently a remarkable
realism, yet synthetic image data generated this way reveals a significant
domain gap with respect to real-world data. This is especially true in
autonomous driving scenarios, which represent a critical aspect for overcoming
utilizing synthetic data for training neural networks. We propose a method
based on domain-invariant scene representation to directly synthesize traffic
scene imagery without rendering. Specifically, we rely on synthetic scene
graphs as our internal representation and introduce an unsupervised neural
network architecture for realistic traffic scene synthesis. We enhance
synthetic scene graphs with spatial information about the scene and
demonstrate the effectiveness of our approach through scene manipulation.

##  I Introduction

A broad variety of real world scenarios require autonomous navigation systems
to rely on machine learning-based perception algorithms. Such algorithms are
knowingly data dependent, yet data acquisition and labeling is a costly and
tedious process. It is associated with manual labor, must handle rare long
tail corner case events, and could be hard constrained by ethical aspects
e.g., in case of near-accident scenarios.

Figure 1: BDD and Cityscapes traffic scenes generated from synthetic scene
graph.

One of the common alternatives to real data acquisition and annotation is
represented by simulation and synthetic data. Simulation has a long history in
driver assistance systems, but with the renaissance of neural networks, the
research community has strengthened efforts in this direction and many
synthetic datasets [28, 27, 34] and simulation systems [9, 30] appeared.

$P$$G$$\mathcal{L}_{SG}$$\mathcal{L}_{TS}$$b_{i},m_{i}$$\hat{b}_{i},\hat{m}_{i}$$x$$\hat{x}$
Figure 2: Overview of the network with graph processor $P$, traffic scene
generator $G$, and loss functions $\mathcal{L}_{SG}$ and $\mathcal{L}_{TS}$.

Despite being a powerful research tool, synthetic data typically reveals a
significant domain gap with respect to target real data. The underlying
phenomenon, where marginal distributions in both domains differ, has been
defined by [32] as a covariate shift. This means that i.i.d assumption does
not hold for a synthetic-real setup. Simply stated, synthetic and real domains
differ both in content and appearance. This problem is normally a subject for
domain adaptation methods such as sim2real domain transfer.

In this work, we propose overcoming the domain gap between both domains by
eliminating the rendering part from the pipeline. The intuition behind this
concept tells that rendered images introduce a bias towards the underlying
domain, which is then commonly leveled out by e.g., style transfer methods.
Our main idea is to replace the rendering with abstract scene representation
and directly synthesize realistic images of traffic scenes from it. We utilize
scene graphs as such abstract representations. Scene graphs encode objects in
the scene as nodes and relations between them as edges [18], in addition, they
can also integrate certain characteristics of the objects as attributes [2].

In this regard, scene graphs are domain agnostic as they can be generated in
simulation and be applied to real-world data without a strong domain gap. In
fact, scene graphs are fairly simple to simulate and manipulate, which allows
for domain randomization and data generation of potentially arbitrary size and
variance.

Derived from a simulation, synthetic scene graphs incorporate relevant objects
as car, person etc. and relations like left to, right to. We additionally
extend this setup with necessary traffic scene classes like road, sidewalk,
building, vegetation. It is important to note that simulation provides 3D
information about the scene, which can be introduced into the scene graph in
the form of spatial attributes and spatial relations between objects.
Moreover, synthetic annotations provided by simulation are pixel precise and
could be used in a downstream task.

In this work, we propose synthetic 3D scene graphs with spatial components. We
also provide the aforementioned graph representations for existing synthetic
urban traffic datasets PfB [27] and Synscapes [34]. To enable realistic
traffic scene generation from synthetic scene graphs, we propose a neural
network architecture that supports unsupervised training. We demonstrate the
benefit of our approach with an online tool through traffic scene generation
and manipulation: Demo.

##  II Related Work

Rendering. Historically, traffic scene synthesis was commonly achieved through
computer graphics. Multiple works focused on physically realistic rendering of
urban traffic environments: SYNTHIA [28], Virtual KITTI [11], PfB [27],
VKITTI2 [4]. The most recent and visually realistic one is arguably Synscapes
[34]. Simulation can be applied not only in outdoor environments, but also in
indoor scene synthesis for autonomous agents [21], [26]. Although rendered
data reveals a high level of realism and a great deal of variations, it is
still affected by a significant domain gap with respect to real data when it
is used for training machine learning approaches, in particular, neural
networks.
Domain Transfer. To mitigate the limitation introduced by sim2real domain
shift, research has focused on data synthesis using deep neural networks. The
vast majority of such models [23], [13], [3], [19], utilizes adversarial
frameworks (GAN) [12] for image generation. Many of them condition the
generation process on visual artifacts or descriptions derived from available
labels (semantics, edges etc.), such as Pix2Pix [16], CRN [5], Pix2PixHD [33]
and SPADE [25]. There are also several works which employed unsupervised
domain adaptation from synthetic to real images, such as CycleGAN [39, 1],
DIRT-T [31], MUNIT [15].
Scene Graphs. Another relevant research direction is represented by the use of
more domain invariant scene descriptions such as text [38] or scene graphs.
Scene graphs are abstract data structures used for describing scenes by
encoding objects as nodes and relations between them as edges. Thus, the whole
scene can be represented as a directed graph. Scene graphs have been used as
an alternative to natural language description for image retrieval [18] and
image description [22]. Most recent works on scene graphs for image generation
include [17], [2] and [8]. [2] proposed dual layout and appearance embedding
for better matching of generated scenes and underlying scene graphs. [8]
utilizes scene graphs as an intermediate representation for image manipulation
without direct supervision. A lot of works in the area of scene graphs rely on
the Visual Genome dataset [20], which provides image samples annotated with
scene graphs. To our knowledge, there is no such dataset available for urban
traffic environments, so we provide scene graph annotations for commonly used
public datasets, in particular, PfB [27] and Synscapes [34].

Figure 3: Examples of synthetic scene graphs and corresponding generated
traffic scenes from Cityscapes.

Figure 4: Examples of synthetic scene graphs with corresponding BDD generated
traffic scenes and semantic maps.

##  III Proposed Approach

Our approach is focused on synthetic scene graphs and unsupervised traffic
scene generation, where pairs of synthetic and real scenes are not available.
We derive a synthetic scene graph from a procedurally generated scene that
does not provide visual characteristics such as textures or materials. For the
synthesized scene graph, we then produce an image of the corresponding traffic
scene, which resembles the content of the underlying synthetic scene and the
realistic appearance of the target data. Figure 2 provides an overview of our
method. It highlights the two main parts of the approach: synthetic scene
graph generation and realistic image generation.

###  III-A Synthetic 3D Scene Graphs

We adopt the setup from [17] for traffic scene data, where every scene graph
is represented by nodes $n_{i}$ associated to each object $i$ in the scene and
edges $e_{ij}$ encode relations between particular nodes $n_{i}$ and $n_{j}$.
Our graph processor is built upon [2] but extends it in several ways, allowing
3D scene graph construction. In addition to the simplified relations - e.g.,
left of, above [2], we integrate 3D information about the scene in form of
spatial relations (in front of, behind) between objects and spatial attributes
(depth component z) of particular objects. Importantly, this information is
available in simulation at no cost. To generate a comprehensive traffic scene,
we extend the objects list and integrate background classes such as sky,
building, vegetation.

Figure 5: BDD and Cityscapes traffic scenes generated from the same synthetic
scene graph.

Therefore, every node in our graph is represented as
$n_{i}=[o_{i},l_{i},z_{i}],o_{i}\in\mathbb{R},l_{i}\in\\{0,1\\}^{L\times
L},z_{i}\in\\{0,1\\}^{Z}$, where $o_{i}$ takes one of the indexes of $C$
possible object classes, $L$ is size of the grid to denote objects location
$l_{i}$ and $Z$ is the scene depth to denote objects position $z_{i}$ along
the z-axis. Analogously, every edge is represented as $e_{i,j}\in\mathbb{R}$,
taking one of the values from predefined dictionary of relation types.

To train the graph processor, we also obtain mask $m_{i}$ and the bounding box
$b_{i}\in\mathbb{R}^{4}$ for each object $i$ from the simulation. Hence, our
graph processor $P$ is trained in a supervised manner to produce the full
scene layout $t\in\mathbb{R}^{H\times\ W\times C}$. Similar to [17], our graph
processor $P$ is composed of 3 networks: graph network, mask regression
network and box regression network. The graph network is constructed of graph
convolutional layers which extract features from scene graphs and encode per-
object layout embedding, mask network consists of several deconvolution
layers, and box network is a Multi-Layer Perceptron (MLP). The last two
networks are dedicated to predicting masks and bounding boxes of the objects
present in the scene.

As an output of the first step, we obtain a synthetic scene graph alongside
the corresponding layout $t$ produced by the graph processor $P$. Both scene
graphs and layouts are domain agnostic, therefore, we omit visual information
from the synthetic domain (rendering) and merely encode content.

###  III-B Traffic Scenes

In the next step, we apply the image generator $G$ to layout $t$ to produce a
realistic traffic scene image $x$, which visually resembles the target data
$X=\\{\hat{x}\in\mathcal{X}\\}$ while at the same time preserving the content
of the original scene graph. Since our approach focuses on the realistic image
generation from the synthetic scene graphs, it requires unsupervised training,
as synthetic-real pairs are not available. To enable unsupervised training we
employ adversarial [12] and contrastive [14] losses. Hence, our final
objective is as follows:

$\displaystyle\begin{split}&\mathcal{L}=\mathcal{L}_{SG}+\mathcal{L}_{TS}\\\
&=\mathcal{L}_{MSE}(b_{i},\hat{b}_{i})+\mathcal{L}_{GAN}(m_{i},\hat{m}_{i})+\mathcal{L}_{FM}(m_{i},\hat{m}_{i})\\\
&+\mathcal{L}_{GAN}(x,\hat{x})+\mathcal{L}_{NCE}(x,\hat{x})\\\ \end{split}$
(1)

Where $\mathcal{L}_{FM}$ is a feature matching loss [29], $\mathcal{L}_{NCE}$
is a multilayer, patch-wise contrastive loss [6, 24] and $\mathcal{L}_{FM}$ is
an adversarial loss applied both to masks $m_{i}$ and images $x$. With
$\hat{}$ we denote either a synthetic ground-truth (for masks) or a target
dataset (for images):

$\displaystyle\begin{split}&\mathcal{L}_{GAN}(m_{i},\hat{m}_{i})=\\\
&\mathbb{E}\log D_{m}(m_{i})+\mathbb{E}\log(1-D_{m}(\hat{m}_{i}))\\\
&\mathcal{L}_{FM}(m_{i},\hat{m}_{i})=\left\lVert\mathbb{E}f(m_{i})-\mathbb{E}_{x\sim}f(\hat{m}_{i})\right\rVert_{2}^{2}\\\
&\mathcal{L}_{GAN}(x,\hat{x})=\mathbb{E}_{x\sim X}\log
D(x)+\mathbb{E}_{\hat{x}\sim X}\log(1-D(\hat{x}))\\\
&\mathcal{L}_{NCE}(x,\hat{x})=\mathbb{E}_{x\sim
X}\sum_{l}\sum_{s}\ell(\hat{x}_{l}^{s},x_{l}^{s},\overline{x}_{l}^{s})\end{split}$
(2)

Where $f$ denotes activations on an intermediate layer of the discriminator,
$\ell$ is a cross-entropy loss function for a positive pair of examples and
$x_{l}^{s}$ are the features of the generator from $l$-th layer at $s$-th
location [6].

##  IV Experiments

To properly evaluate our approach, we design the synthetic graph generator to
operate on bounding boxes and semantic as input. This allows producing
synthetic scene graphs for any existing synthetic image dataset, which
provides 3D bounding boxes and semantic segmentation labels. In particular we
use 2 synthetic traffic datasets for scene graph generation: PfB [27] and
Synscapes [34]. Both are large-scale datasets that provide about 25000 urban
traffic environment images at $1914\times 1052$ and $1440\times 720$ alongside
2D/3D bounding boxes, semantic and instance maps.

In the image generation part, we utilize two real datasets, Cityscapes [7] and
Berkley Deep Drive [36] as providers of appearance characteristics. The first
one comprises about 3000 images of traffic scenes from several German cities
with finely annotated semantic maps. The second one includes about 100000
images of the streets of US American cities, of which we only use 10000.

No car

z-attribute 5

z-attribute 2

Figure 6: An example of traffic scene manipulation by changing spatial
attribute of the car object.

No relation

in front of

behind

Figure 7: An example of traffic scene manipulation by changing spatial
relation of 2 car objects.

Building right of the car

Vegetation right of the car

Vegetation above the car

Building above the car

Figure 8: An example of traffic scene manipulation by changing classes. Figure
9: Comparison of the generated images (top$\rightarrow{}$bottom):
CycleGAN[39], DualGAN[35], MUNIT[15], DRIT[31], ours

###  IV-A Synthesis

We first produce a set of synthetic scene graphs that includes 5k samples. We
rely on this set to train our scene graph generator. For both experiments with
Cityscapes and BDD, our setup is similar. We conduct all experiments at the
resolution of $256x\times 512$ pixels. The training continues for 150 epochs
with a learning rate of $1e-3$ and momentum $0.99$. Figures 3 and 4 show the
results for the training on Cityscapes and BDD, respectively. Additionally, in
figure 9 we compare our method with state-of-the-art unsupervised image
translation techniques - e.g., CycleGAN [39], MUNIT [15]. This is also
reflected in the FID score - CycleGAN achieves 103.05, MUNIT - 75.98 and our
method 47.26.

It is evident from the aforementioned figures that the traffic scene generator
picks up the appearance of both datasets nicely, producing in general
realistic images. Being slightly inaccurate in fine details (especially those
of cars), it is yet very consistent on the class and image level. The
generator produces a very congruous appearance of the objects of following
classes: road, sidewalk, building, vegetation. We also want to highlight the
conditioning of the generation process illustrated in Figures 3 and 4 in
sections IV-C and IV-B. There we demonstrate traffic scene manipulation with
regard to newly introduced classes, spatial attributes and spatial relations
to ensure that our improvements take effect.

###  IV-B Spatial Attributes and Relations

Synthetic scene graphs let us leverage spatial information about the traffic
scene. Figure 6 visualizes 3 traffic scenes generated from the same scene
graph by varying the z-component of a particular object in the graph. In the
first image, no car is present in the scene. In the following images, we put a
car in the scene and change its position by manipulating the value of the
z-attribute.

The 3D information for the scene not only enables changing the z-attribute of
the objects, but also enables spatial relation between them. Thus, Figure 7
shows a traffic scene produced by the scene graph by swapping the relation
between two cars, which appear in the scene having approximately the same size
when not bounded by any spatial relation. The right car gets pulled toward the
ego vehicle, while the left car appears farther away from the case of in front
of relation. When the relation changes to behind, the left car is rendered
significantly bigger than before and bigger than the right car.

| Sky | Road | Tree | Building | Person | Car | Bus | Truck | Mean
---|---|---|---|---|---|---|---|---|---
PfB 25k [27] | 62.8 | 41.1 | 67.8 | 64.3 | 14.0 | 38.8 | 1.1 | 8.1 | 37.3
Ours 3D 5k | 23.4 | 76.1 | 44.7 | 46.3 | 8.1 | 41.2 | 3.5 | 1.6 | 30.6
Ours 3D 10k | 34.2 | 68.0 | 62.5 | 60.8 | 9.7 | 40.7 | 1.3 | 3.0 | 35.0
Ours 3D 5k + PfB | 25.4 | 80.2 | 55.9 | 64.6 | 29.3 | 64.2 | 6.3 | 7.7 | 41.7
Ours 3D 2k* | 64.7 | 91.3 | 67.4 | 64.9 | 13.2 | 39.5 | 1.5 | 2.8 | 43.2

TABLE I: Segmentation results of DRN26 network. Training performed on
generated data and evaluation made on Cityscapes val [7] dataset. (*) denotes
class balanced scene graphs.

###  IV-C Traffic Scene Classes

Additionally to objects (car, bus) we introduce multiple classes which are
characteristic for traffic scene scenarios. Such classes include road,
sidewalk, vegetation, building, sky. To verify that introducing background
classes is effective, we perform several experiments by manipulating the
classes in the traffic scenes. Figure 8 shows the effects of such
manipulation: changing classes of the particular nodes in the scene graph
results in the respective adjustment of the semantic layout of the generated
scene.

In addition to, to a qualitative evaluation of the approach, we conduct
several experiments on the downstream task of semantic segmentation. This
provides a quantitative assessment of the data generated by our method.

###  IV-D Image Segmentation

To assess the proposed method quantitatively, we train a state-of-the-art
semantic segmentation method on the data produced by our method as well as on
the underlying synthetic data. Here we focus on the scene graphs produced from
the Cityscapes simulation - i.e, we randomly generate 2 sets of scene graphs
with 5000 and 10000 samples. For these scene graphs, we generate corresponding
images alongside semantic maps. They are then used to train DRN [37].

We train a segmentation network for 200 epochs on 8 classes of interest and
evaluate on the real Cityscapes val dataset. We provide per class IoU score
and meanIoU also referred as Jaccard Index [10] in Table I. The table
demonstrates that DRN trained on the generated 5k dataset lies 5% behind DRN
trained on the original simulated PFD 25k dataset. Doubling the number of
scene graphs and corresponding images reduces the gap to 2 points.
Additionally, we report in the table I the results of the experiment where
scene graphs follow the statistics of a real dataset. Knowing the target data
class ratio allows us to sub-sample the data and keep those synthetic scene
graphs whose layouts follow target data class ratio. This makes it possible to
reduce the scene graph number to 2000 samples and improve segmentation
performance by 5% compared to the original 25k dataset.

##  V Conclusion

In this work, we propose a method to ease data generation for realistic
traffic scenes from domain agnostic scene representation called scene graphs
instead of using photo-realistic rendering. We utilize synthetic scene graphs
enhanced by spatial attributes (z) and spatial relations (e.g., behind).
Furthermore, we introduce an unsupervised approach for realistic image
generation from synthetic scene graphs. The approach shows convincing
generation results as demonstrated in the proposed qualitative evaluation. We
also show the effectiveness of our method through traffic scene manipulation
and validation on a downstream task.

##  VI Acknowledgment

The research leading to these results is funded by the German Federal Ministry
for Economic Affairs and Energy within the project “KI Absicherung – Safe AI
for Automated Driving”. The authors would like to thank the consortium for the
successful cooperation.

## References

  * [1] A. Almahairi, S. Rajeshwar, A. Sordoni, P. Bachman, and A. Courville.  Augmented CycleGAN: Learning many-to-many mappings from unpaired data.  In International Conference on Machine Learning, 2018.
  * [2] O. Ashual and L. Wolf.  Specifying object attributes and relations in interactive scene generation.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [3] A. Brock, J. Donahue, and K. Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In International Conference on Learning Representations, 2019.
  * [4] Y. Cabon, N. Murray, and M. Humenberger.  Virtual kitti 2, 2020.
  * [5] Q. Chen and V. Koltun.  Photographic image synthesis with cascaded refinement networks.  In IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In International Conference on Machine Learning, 2020.
  * [7] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The cityscapes dataset for semantic urban scene understanding.  In IEEE CVPR, 2016.
  * [8] H. Dhamo, A. Farshad, I. Laina, N. Navab, G. D. Hager, F. Tombari, and C. Rupprecht.  Semantic image manipulation using scene graphs.  In IEEE CVPR, 2020.
  * [9] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun.  CARLA: An open urban driving simulator.  In Annual Conference on Robot Learning, 2017.
  * [10] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  International Journal of Computer Vision, 2015.
  * [11] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig.  Virtual worlds as proxy for multi-object tracking analysis.  In IEEE conference on Computer Vision and Pattern Recognition, 2016\.
  * [12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, 2014.
  * [13] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville.  Improved training of wasserstein gans.  In Advances in Neural Information Processing Systems, 2017.
  * [14] M. Gutmann and A. Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In International Conference on Artificial Intelligence and Statistics, 2010.
  * [15] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz.  Multimodal unsupervised image-to-image translation.  In ECCV, 2018.
  * [16] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [17] J. Johnson, A. Gupta, and L. Fei-Fei.  Image generation from scene graphs.  In IEEE CVPR, 2018.
  * [18] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei.  Image retrieval using scene graphs.  In IEEE Conference on Computer Vision and Pattern Recognition, 2015\.
  * [19] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, 2020.
  * [20] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 2017.
  * [21] J. McCormac, A. Handa, S. Leutenegger, and A. J. Davison.  Scenenet rgb-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation?  In ICCV, 2017.
  * [22] A. Newell and J. Deng.  Pixels to graphs by associative embedding.  In Advances in Neural Information Processing Systems, 2017.
  * [23] A. Odena, C. Olah, and J. Shlens.  Conditional image synthesis with auxiliary classifier GANs.  In International Conference on Machine Learning, 2017.
  * [24] T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu.  Contrastive learning for unpaired image-to-image translation.  In European Conference on Computer Vision, 2020.
  * [25] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [26] W. Qiu, F. Zhong, Y. Zhang, S. Qiao, Z. Xiao, T. S. Kim, Y. Wang, and A. Yuille.  Unrealcv: Virtual worlds for computer vision.  ACM Multimedia Open Source Software Competition, 2017.
  * [27] S. R. Richter, Z. Hayder, and V. Koltun.  Playing for benchmarks.  In IEEE International Conference on Computer Vision, 2017.
  * [28] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In CVPR, 2016.
  * [29] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen.  Improved techniques for training gans.  In NIPS, 2016.
  * [30] S. Shah, D. Dey, C. Lovett, and A. Kapoor.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles.  In Field and Service Robotics, 2017.
  * [31] R. Shu, H. H. Bui, H. Narui, and S. Ermon.  A dirt-t approach to unsupervised domain adaptation.  In ICLR, 2018.
  * [32] M. Sugiyama and M. Kawanabe.  Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation.  The MIT Press, 2012.
  * [33] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In IEEE/CVF CVPR, 2018.
  * [34] M. Wrenninge and J. Unger.  Synscapes: A photorealistic synthetic dataset for street scene parsing.  In https://arxiv.org/abs/1810.08705, 2020.
  * [35] Z. Yi, H. Zhang, P. Tan, and M. Gong.  Dualgan: Unsupervised dual learning for image-to-image translation.  In ICCV, 2017.
  * [36] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In https://arxiv.org/abs/1810.08705, 2018.
  * [37] F. Yu and V. Koltun.  Multi-scale context aggregation by dilated convolutions.  In International Conference on Learning Representations (ICLR), 2016\.
  * [38] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In ICCV, 2017.
  * [39] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE International Conference on Computer Vision (ICCV), 2017\.


# Resolving XENON Excess With Decaying Cold Dark Matter

Shuai Xu<EMAIL_ADDRESS>Department of Physics, Chongqing University,
Chongqing 401331, China  Sibo Zheng<EMAIL_ADDRESS>Department of
Physics, Chongqing University, Chongqing 401331, China

###### Abstract

We propose a decaying cold dark matter model to explain the excess of electron
recoil observed at the XENON1T experiment. In this scenario, the daughter dark
matter from the parent dark matter decay easily obtains velocity large enough
to saturate the peak of the electron recoil energy around 2.5 keV, and the
observed signal rate can be fulfilled by the parent dark matter with a mass of
order $10-200$ MeV and a lifetime larger than the age of Universe. We verify
that this model is consistent with experimental limits from dark matter
detections, Cosmic Microwave Background and Large Scale Structure experiments.

##  I Introduction

Recently, XENON1T experiment Aprile:2020tmw , which is a dark matter (DM)
direct detection facility, has reported an excess of electron recoil over the
background in the $1-7$ keV range with $3.5\sigma$ significance. As pointed
out in ref.Aprile:2020tmw , this excess is unlikely due to solar axion,
neutrino magnetic moment or statistical uncertainties about the background. So
far, the observed excess has initiated extensive investigations about
potential astrophysical sources. Among other things, a cold DM is a natural
candidate, which is the subject of this study.

To explain the observed excess, one has to accommodate two critical quantities
\- the electron recoil energy range around 2-3 keV and the electron transfer
momentum range near 50 keV. Unfortunately, they conflict with a conventional
cold DM, which has a velocity typically of order $\sim 10^{-3}c$, with $c$ the
velocity of light. A few proposals have been proposed to avoid the conflicts.
In the case of elastic scattering Kannike:2020agf ; Fornal:2020npv ;
Su:2020zny ; Bell:2020bes ; Cao:2020bwd ; Jho:2020sku , the cold DMs can be
boosted in certain circumstances before they interact with the electrons in
the xenon atoms, while in the case of inelastic scattering Harigaya:2020ckz ;
Bramante:2020zos ; Aboubrahim:2020iwb ; Lee:2020wmh ; Baryakhtar:2020rwy  the
favored electron transfer momentum range can be realized in terms of small
rest mass splitting between two different DM components.

In this study, we propose a new decaying cold DM scenario, in which the parent
DM (A) decays to the daughter DM particles (B),

$\displaystyle{}A\rightarrow BB.$ (1)

In terms of the decay, the velocity of the daughter DM can be enhanced to be
comparable with $c$, relative to the small velocity of the parent DM. Unlike
photoelectric absorption in a decaying warm DM Choi:2020udy , the daughter
particle elastically scatters off the electrons in the xenon atoms. We will
show that without any violations of current DM (in)direct detections or
cosmological measurements, this decaying cold DM model can easily resolve the
XENON excess.

##  II The Model

We begin with the production of the daughter particle $B$ due to $A$ decay.
The decay yields the following velocity and present number density of $B$
respectively,

$\displaystyle{}\frac{\upsilon_{B}}{c}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{m^{2}_{A}}{4m^{2}_{B}}-1},$ (2) $\displaystyle
n_{B}$ $\displaystyle=$ $\displaystyle
2\frac{\rho_{\text{dm},0}}{m_{A}}\left[1-\exp{(-\Gamma_{A}\mathrm{t}_{0})}\right],$
(3)

where $m_{A}$ and $m_{B}$ refer to the parent and daughter DM mass
respectively with $m_{A}>2m_{B}$, $\mathrm{t}_{0}$ is the age of Universe,
while $\rho_{\text{dm},0}=0.4~{}\mathrm{GeV}/\text{cm}^{3}$ and $\Gamma_{A}$
denote the local DM density and the decay width of the parent DM $A$,
respectively. We will assume that compared to the decay production the thermal
production of $n_{B}$ is subdominant.

The input parameters in Eqs.(2)-(3) are constrained as follows. Firstly, in
order to yield $\upsilon_{B}$ of order $\sim 0.1~{}c$ the mass ratio
$m_{A}/2m_{B}$ should deviate from unity in percent level, which implies that
$m_{A}$ and $2m_{B}$ are highly degenerate. Secondly, in order to fulfill the
cosmological bounds on the decaying DM both from the Cosmic Microwave
Background (CMB) and Large Scale Structure (LSS) experiments, the lifetime of
$A$ particle $\tau_{A}=\Gamma^{-1}_{A}$ should be larger than
$\mathrm{t}_{0}$, which can be achieved by adjusting the coupling constant
between $A$ and $B$, with the help of a suppression by the small $\beta$
factor due to the mass degeneracy.

We will return to the cosmological constraints after we have explored the
signal rate of the recoil electrons at the XENON1T.

##  III Signal Rate

According to conservations of energy and momentum in the elastic scattering
process, the energy111In the case of inelastic scattering, $E_{e}$ in Eq.(4)
receives a new term due to the rest mass splitting between two different
components that involve in the scattering off electron. transferred to
electron reads Bloch:2020uzh

$\displaystyle{}E_{e}=\mathbf{q}\cdot\overrightarrow{\upsilon}_{B}-\frac{q^{2}}{2m_{B}}.$
(4)

From Eq.(4) the maximal value
$E_{e}^{\mathrm{max}}\approx\frac{1}{2}m_{B}\upsilon^{2}_{B}$ at $q\approx
m_{B}\upsilon_{B}$ valid only when $m_{B}\approx m_{e}$. Consider that when
$m_{B}\leq m_{e}$ the daughter DM with a large velocity is severely
constrained by limits such as the effective number of neutrinos, we will focus
on $m_{B}\gg m_{e}$, under which $E_{e}^{\mathrm{max}}\approx
2m_{e}\upsilon^{2}_{B}$ Kannike:2020agf  instead. This constraint implies
$\upsilon_{B}\geq 0.05~{}c$ in order to satisfy $E_{e}^{\mathrm{max}}\geq 2.5$
keV.

Given a fixed value of $E_{e}$, Eq.(4) determines the electron transfer
momentum range $q_{-}<q<q_{+}$, with

$\displaystyle{}q_{\pm}=m_{B}\upsilon_{B}\pm\sqrt{m^{2}_{B}\upsilon^{2}_{B}-2m_{B}E_{e}}.$
(5)

The transfer momentum range in Eq.(5) affects the signal rate of the recoil
electrons discussed below in the sense that the atomic factorization factor
$K(E_{e},q)$ Roberts:2019chv ; Roberts:2016xfw  is rather sensitive to $q$,
which takes the maximal value $K_{\mathrm{max}}\approx 0.1$ at
$q_{\mathrm{peak}}\approx 50$ keV for $E_{e}=2$ keV, and dramatically declines
as $q$ slightly deviates from $q_{\mathrm{peak}}$. Therefore, in order to
maximize the $K$-factor contribution to the signal rate, we should take
suitable values of $m_{B}$ and $\upsilon_{B}$ to make sure that
$q_{\mathrm{peak}}$ is covered by the electron transfer momentum range in
Eq.(5).

Furthermore, the daughter DM-free electron scattering cross section
$\bar{\sigma}_{e}$ relies on the nature of mediator Alhazmi:2020fju  which
communicates the interaction between the daughter DM B and electron. From the
viewpoint of effective field theory, $\bar{\sigma}_{e}$ can be written as

$\displaystyle{}\bar{\sigma}_{e}\approx\frac{g^{2}_{mB}g^{2}_{me}m^{2}_{e}}{\pi
m^{4}_{\mathrm{med}}},$ (6)

where $m_{\mathrm{med}}$ is the mediator mass, $g_{mB}$ is the coupling
between the mediator and $B$, and $g_{me}$ is the coupling between the
mediator and electron. If the mediator is identified as a standard model
particle, only $g_{mB}$ in Eq.(6) is a free parameter, the magnitude of which
has to be constrained by the decay width of the standard model particle.

After a handle on the “luminosity” and the DM B-electron scattering cross
section, we now estimate the number of events of recoil electrons

$\displaystyle{}\frac{dR}{dE}$ $\displaystyle\approx$ $\displaystyle
n_{\mathrm{xe}}n_{B}\times\frac{\bar{\sigma}_{e}}{2m_{e}\upsilon_{B}}$
$\displaystyle\times$ $\displaystyle\int
dE_{e}\left[\int^{q_{+}}_{q_{-}}dqa^{2}_{0}q\mid
F(q)\mid^{2}K(E_{e},q)\right]R_{s}(E,E_{e}),$

where $n_{\mathrm{xe}}\approx 4.2\times 10^{27}/$ton is the number density of
xenon atoms in the detector, $a_{0}=1/(\alpha_{\mathrm{em}}m_{e})$ is the Bohr
radius with $\alpha_{\mathrm{em}}=1/137$, $F(q)\approx 1$ is the DM form
factor, and $R_{s}$ is the resolution function which accounts for the
“efficiency” of the detector. We will simply take the Gaussian distribution
for the reconstructed energy for numerical analysis

$\displaystyle{}R_{s}(E,E_{e})=\frac{\alpha(E)}{\sqrt{2\pi}\sigma}\exp\left[-\frac{(E-E_{e})^{2}}{2\sigma^{2}}\right],$
(8)

where $\alpha(E)$ is the efficiency Aprile:2020tmw  and
$\sigma=a\sqrt{E_{e}}+bE_{e}$, with $a=(0.310\pm 0.004)$ $\sqrt{\mathrm{keV}}$
and $b=0.0037\pm 0.0003$, respectively.

Figure 1: Fit to the observed data Aprile:2020tmw  about the number of events $dR/dE$ $(\mathrm{ton~{}year~{}keV})^{-1}$ as function of the reconstructed electron recoil energy with five different sets of benchmark values C1 to C5 in Table.1, where $B_{0}$ and $B$ represents the background and the daughter DM contribution, respectively. |  $m_{A}$ (MeV) |  $m_{B}$(MeV) | $\upsilon_{B}/c$ | $\tau_{A}/\mathrm{t}_{0}$ |  $\bar{\sigma}_{e}$ $(\mathrm{cm}^{2})$
---|---|---|---|---|---
C1 | 10.05 | 5 | 0.1 | $3$ | $4.36\times 10^{-45}$
C2 | 20.1 | 10 | 0.1 | $3$ | $9.46\times 10^{-45}$
C3 | 40.2 | 20 | 0.1 | $3$ | $1.89\times 10^{-44}$
C4 | 100.5 | 50 | 0.1 | $3$ | $4.22\times 10^{-44}$
C5 | 201 | 100 | 0.1 | $3$ | $8.06\times 10^{-44}$
Table 1: Five sets of benchmark values which yield the same fit as shown in
Fig.1, where the required values of $\bar{\sigma}_{e}$ can be understood as an
output parameter.

Fig.1 shows the fit to the reported XENON1T data Aprile:2020tmw  with five
different sets of benchmark values C1 to C5 as explicitly shown in Table.1. In
individual case therein, we have chosen fixed value
$\tau_{A}=3~{}\mathrm{t}_{0}$, under which $m_{A}\approx 2m_{B}$ take the mass
ranges of $10-200$ MeV and $\upsilon_{B}/c=0.1$. The values of
$\bar{\sigma}_{e}$ inferred from the observed XENON excess varies from
$\mathcal{O}(10^{-45})$ cm2 to $\mathcal{O}(10^{-44})$ cm2.

##  IV Dark Matter Constraints

Now we turn to possible constraints on the dark matter particles $A$ and $B$.
Since the interaction in Eq.(1) yields too small annihilation cross section
for $A$ to accommodate the required thermal annihilation cross section, $A$
has to communicate either with the Standard Model (SM) sector e.g. via the
same mediator as $B$, or mainly with other unstable freedoms in the dark
sector. In the former situation, some constraints on $B$ as below can be
placed on $A$ as well.

With the communication between $B$ and the SM sector as inferred from the
XENON1T excess, we can at least place the following constraints.

  * •

The daughter DM B-free electron scattering cross section $\bar{\sigma}_{e}$,
extracted from the XENON1T excess, can be used to constrain the model
parameters. Based on the measurements on $\bar{\sigma}_{e}$ within various
electron recoil energies, the light daughter DM can be probed either by the
current XENON1T Aprile:2019xxb ; Essig:2017kqs  or the future SuperCDMS
Essig:2015cda  experiments.

  * •

Similar to the DM B-electron scattering, we can also constrain $m_{B}$ from
the annihilation cross section $\sigma^{B}_{\mathrm{ann}}(BB\rightarrow
e^{+}e^{-})$, based on the cross symmetry between the two Feynman diagrams
related to these two processes. While experiments such as AMS-01
Aguilar:2007yf , AMS-02 Aguilar:2013qda  or PAMELA Adriani:2013uda  have not
yet placed viable bounds on $\sigma^{B}_{\mathrm{ann}}$ in the sub GeV-scale
DM mass, the Planck data Ade:2015xua  is able to constrain $m_{B}$ down to
$\sim 1$ MeV.

  * •

Lastly, the coupling of the mediator to electron can be constrained by
colliders such as BaBar, LEP and LHC.

For illustration, we show in Fig.2 the constraints in the specific dark photon
model with the mediator identified as a new vector boson $A^{\prime}$, where
$m_{A^{\prime}}=1$ GeV and $g_{me}=5\times 10^{-4}e$ have been adopted in the
light of BaBar data Lees:2014xha . In this figure, we have simultaneously
shown the SuperCDMS limit Essig:2015cda  (in blue) without relativistic effect
Pandey:2018esq , the Planck 2015 limit Slatyer:2015jla  (in red) and the
parameter space (in black curve) together with the benchmark values in
Table.1. Relatively weaker XENON1T limit has been ignored. One observes that
in this explicit model $m_{B}$ beneath $\sim 15$ MeV survives.

Figure 2: Constraints on the parameter space (in black curve) in the dark
photon model with $m_{A^{\prime}}=1$ GeV, $g_{me}=5\times 10^{-4}e$ and the DM
form factor $F(q)=1$. We have simultaneously shown the SuperCDMS limit
Essig:2015cda  (in blue) and the Planck 2015 limit Slatyer:2015jla  (in red),
where the shaded regions are excluded.

Compared to the benchmark values in Table.1, given fixed $m_{B}$ one can
obtain larger $\bar{\sigma}_{e}$ or alternatively larger $g_{mB}g_{me}$ by
taking larger $\tau_{A}$, since they are linearly correlated to each other in
$dR/dE\sim\bar{\sigma}_{e}(\mathrm{t}_{0}/\tau_{A})$ for
$\mathrm{t}_{0}\ll\tau_{A}$ in Eq.(III). However, an increase of $\tau_{A}$
will simultaneously lead to linearly enhanced experimental limits in Fig.2.
These trends together imply that adjusting $\tau_{A}$ is unable to alter the
SuperCDMS sensitivity as illustrated in Fig.2.

##  V Cosmological Constraints

The decaying DM model are constrained both by the CMB and the LSS experiments
for a varying dark matter energy density with time. In our scenario, it reads
from Eq.(3)

$\displaystyle{}\rho_{\mathrm{dm}}(t)=\rho_{\mathrm{dm},0}\left[e^{-t/\tau_{A}}+\frac{2m_{B}}{m_{A}}\left(1-e^{-t/\tau_{A}}\right)\right]a^{-3}(t),$

Compared to the baseline $\Lambda$CDM cosmology, the DM relic density in
Eq.(V) is altered by a magnitude of order
$\mid\Delta\rho_{\mathrm{dm}}/\rho_{\mathrm{dm},0}\mid\approx(1-\frac{2m_{B}}{m_{A}})t/\tau_{A}<10^{-3}$
in the small redshift region for the benchmark values in Table.1, as a result
of highly degenerate dark matter mass relation $m_{A}\approx 2m_{B}$ required
by the XENON excess.

For the CMB experiment Aghanim:2018eyx , it mainly affects the temperature
power spectrum $C_{TT}$ in terms of the integrated Sachs-Wolfe effect, which
relies on the cosmological evolution of Universe after the last scattering.
Due to the small fraction in $\rho_{\mathrm{dm}}$ given by Eq.(V) relative to
what attempts to explain the Hubble tension Vattis:2019efj ;
Anchordoqui:2015lqa ; Bringmann:2018jpr ; Clark:2020miy , the effect on
$C_{TT}$ in our scenario is negligible. For the LSS experiments, the DM power
spectrum $\delta=\delta\rho_{\mathrm{dm}}/\rho_{\mathrm{dm}}$ evolves with
time as

$\displaystyle{}\ddot{\delta}+2H\dot{\delta}-4\pi
G\rho_{\mathrm{dm}}\delta=0,$ (10)

where $G$ is the Newton’s constant and $H$ is the Hubble rate. The small
fraction in $\rho_{\mathrm{dm}}$ gives rise to a fraction in the DM power
spectrum $\delta$ less than the order of a percent level, which is far beyond
the reach of future LSS experiments such as the Dark Energy Spectroscopic
Instrument Aghamousa:2016zmz .

##  VI Conclusions

In this study we have proposed a novel decaying cold DM scenario in which the
cold parent DM $A$ decays to the daughter particle $B$, with the lifetime
$\tau_{A}$ larger than the age of Universe. Firstly we have shown that in this
scenario the observed excess of the electron recoil at the XENON1T in the
energy range $2-3$ keV can be addressed by the daughter DM B-electron elastic
scattering with the DM mass ranges $m_{A}\approx 2m_{B}\sim 10-200$ MeV.
Moreover, we have verified that because of small DM B-electron scattering
cross section this model is consistent with limits both from the DM direct and
indirect detections, while as a result of suppression on the magnitude of the
fraction in the DM energy density due to the highly degenerate mass relation
imposed by the XENON excess, this model does not violate either the CMB
measurements on the temperature power spectrum or the LSS constraints on the
DM power spectrum. Finally, there are a few directions in our DM scenario
which deserve further investigation. Especially, if we are allowed to adopt
the lifetime of the parent DM obviously smaller than the age of Universe, it
is not unlikely to resolve the XENON1T excess and the Hubble tension
simultaneously with a decaying cold DM.

## Acknowledgments

The research is supported in part by the National Natural Science Foundation
of China with Grant No. 11775039 and the Fundamental Research Funds for the
Central Universities at Chongqing University with Grant No. cqu2017hbrc1B05.

## References

  * (1) E. Aprile et al. [XENON], “Excess electronic recoil events in XENON1T,” Phys. Rev. D 102 (2020) no.7, 072004, [arXiv:2006.09721 [hep-ex]].
  * (2) K. Kannike, M. Raidal, H. Veermäe, A. Strumia and D. Teresi, “Dark Matter and the XENON1T electron recoil excess,” Phys. Rev. D 102, no.9, 095002 (2020), [arXiv:2006.10735 [hep-ph]].
  * (3) B. Fornal, P. Sandick, J. Shu, M. Su and Y. Zhao, “Boosted Dark Matter Interpretation of the XENON1T Excess,” Phys. Rev. Lett. 125 (2020) no.16, 161804, [arXiv:2006.11264 [hep-ph]].
  * (4) L. Su, W. Wang, L. Wu, J. M. Yang and B. Zhu, “Atmospheric Dark Matter and Xenon1T Excess,” [arXiv:2006.11837 [hep-ph]].
  * (5) N. F. Bell, J. B. Dent, B. Dutta, S. Ghosh, J. Kumar and J. L. Newstead, “Explaining the XENON1T excess with Luminous Dark Matter,” Phys. Rev. Lett. 125 (2020) no.16, 161803, [arXiv:2006.12461 [hep-ph]].
  * (6) Q. H. Cao, R. Ding and Q. F. Xiang, “Exploring for sub-MeV Boosted Dark Matter from Xenon Electron Direct Detection,” [arXiv:2006.12767 [hep-ph]].
  * (7) Y. Jho, J. C. Park, S. C. Park and P. Y. Tseng, “Leptonic New Force and Cosmic-ray Boosted Dark Matter for the XENON1T Excess,” Phys. Lett. B 811, 135863 (2020), [arXiv:2006.13910 [hep-ph]].
  * (8) K. Harigaya, Y. Nakai and M. Suzuki, “Inelastic Dark Matter Electron Scattering and the XENON1T Excess,” Phys. Lett. B 809, 135729 (2020), [arXiv:2006.11938 [hep-ph]].
  * (9) J. Bramante and N. Song, “Electric But Not Eclectic: Thermal Relic Dark Matter for the XENON1T Excess,” Phys. Rev. Lett. 125 (2020) no.16, 161805, [arXiv:2006.14089 [hep-ph]].
  * (10) A. Aboubrahim, M. Klasen and P. Nath, “Xenon-1T excess as a possible signal of a sub-GeV hidden sector dark matter,” [arXiv:2011.08053 [hep-ph]].
  * (11) H. M. Lee, “Exothermic Dark Matter for XENON1T Excess,” [arXiv:2006.13183 [hep-ph]].
  * (12) M. Baryakhtar, A. Berlin, H. Liu and N. Weiner, “Electromagnetic Signals of Inelastic Dark Matter Scattering,” [arXiv:2006.13918 [hep-ph]].
  * (13) G. Choi, M. Suzuki and T. T. Yanagida, Phys. Lett. B 811, 135976 (2020), [arXiv:2006.12348 [hep-ph]].
  * (14) I. M. Bloch, A. Caputo, R. Essig, D. Redigolo, M. Sholapurkar and T. Volansky, “Exploring New Physics with O(keV) Electron Recoils in Direct Detection Experiments,” [arXiv:2006.14521 [hep-ph]].
  * (15) B. M. Roberts and V. V. Flambaum, “Electron-interacting dark matter: Implications from DAMA/LIBRA-phase2 and prospects for liquid xenon detectors and NaI detectors,” Phys. Rev. D 100, no.6, 063017 (2019), [arXiv:1904.07127 [hep-ph]].
  * (16) B. M. Roberts, V. A. Dzuba, V. V. Flambaum, M. Pospelov and Y. V. Stadnik, “Dark matter scattering on electrons: Accurate calculations of atomic excitations and implications for the DAMA signal,” Phys. Rev. D 93, no.11, 115037 (2016), [arXiv:1604.04559 [hep-ph]].
  * (17) H. Alhazmi, D. Kim, K. Kong, G. Mohlabeng, J. C. Park and S. Shin, “Implications of the XENON1T Excess on the Dark Matter Interpretation,” [arXiv:2006.16252 [hep-ph]].
  * (18) E. Aprile et al. [XENON], Phys. Rev. Lett. 123, no.25, 251801 (2019), [arXiv:1907.11485 [hep-ex]].
  * (19) R. Essig, T. Volansky and T. T. Yu, “New Constraints and Prospects for sub-GeV Dark Matter Scattering off Electrons in Xenon,” Phys. Rev. D 96, no.4, 043017 (2017), [arXiv:1703.00910 [hep-ph]].
  * (20) R. Essig, M. Fernandez-Serra, J. Mardon, A. Soto, T. Volansky and T. T. Yu, “Direct Detection of sub-GeV Dark Matter with Semiconductor Targets,” JHEP 05, 046 (2016), [arXiv:1509.01598 [hep-ph]].
  * (21) M. Aguilar et al. [AMS 01], “Cosmic-ray positron fraction measurement from 1 to 30-GeV with AMS-01,” Phys. Lett. B 646, 145-154 (2007), [arXiv:astro-ph/0703154 [astro-ph]].
  * (22) M. Aguilar et al. [AMS], “First Result from the Alpha Magnetic Spectrometer on the International Space Station: Precision Measurement of the Positron Fraction in Primary Cosmic Rays of 0.5–350 GeV,” Phys. Rev. Lett. 110, 141102 (2013).
  * (23) O. Adriani et al. [PAMELA], “Cosmic-Ray Positron Energy Spectrum Measured by PAMELA,” Phys. Rev. Lett. 111, 081102 (2013), [arXiv:1308.0133 [astro-ph.HE]].
  * (24) P. A. R. Ade et al. [Planck], “Planck 2015 results. XIII. Cosmological parameters,” Astron. Astrophys. 594 (2016), A13, [arXiv:1502.01589 [astro-ph.CO]].
  * (25) J. P. Lees et al. [BaBar], “Search for a Dark Photon in $e^{+}e^{-}$ Collisions at BaBar,” Phys. Rev. Lett. 113 (2014) no.20, 201801, [arXiv:1406.2980 [hep-ex]].
  * (26) M. K. Pandey, L. Singh, C. P. Wu, J. W. Chen, H. C. Chi, C. C. Hsieh, C. P. Liu and H. T. Wong, “Constraints on spin-independent dark matter scattering off electrons with germanium and xenon detectors,” [arXiv:1812.11759 [hep-ph]].
  * (27) T. R. Slatyer, “Indirect dark matter signatures in the cosmic dark ages. I. Generalizing the bound on s-wave dark matter annihilation from Planck results,” Phys. Rev. D 93 (2016) no.2, 023527, [arXiv:1506.03811 [hep-ph]].
  * (28) N. Aghanim et al. [Planck], “Planck 2018 results. VI. Cosmological parameters,” Astron. Astrophys. 641, A6 (2020), [arXiv:1807.06209 [astro-ph.CO]].
  * (29) K. Vattis, S. M. Koushiappas and A. Loeb, “Dark matter decaying in the late Universe can relieve the H0 tension,” Phys. Rev. D 99, no.12, 121302 (2019), [arXiv:1903.06220 [astro-ph.CO]].
  * (30) L. A. Anchordoqui, V. Barger, H. Goldberg, X. Huang, D. Marfatia, L. H. M. da Silva and T. J. Weiler, “IceCube neutrinos, decaying dark matter, and the Hubble constant,” Phys. Rev. D 92, no.6, 061301 (2015), [arXiv:1506.08788 [hep-ph]].
  * (31) T. Bringmann, F. Kahlhoefer, K. Schmidt-Hoberg and P. Walia, “Converting nonrelativistic dark matter to radiation,” Phys. Rev. D 98, no.2, 023543 (2018), [arXiv:1803.03644 [astro-ph.CO]].
  * (32) S. J. Clark, K. Vattis and S. M. Koushiappas, “CMB constraints on late-universe decaying dark matter as a solution to the $H_{0}$ tension,” [arXiv:2006.03678 [astro-ph.CO]].
  * (33) A. Aghamousa et al. [DESI], “The DESI Experiment Part I: Science,Targeting, and Survey Design,” [arXiv:1611.00036 [astro-ph.IM]].


# Learning Deep Video Stabilization without Optical Flow

Muhammad Kashif Ali Equal contribution. Sangjoon Yu11footnotemark: 1 Tae Hyun
Kim
Dept. of Computer Science Corresponding author Hanyang University  Seoul
Korea
<EMAIL_ADDRESS>{kiddyu1991<EMAIL_ADDRESS>
###### Abstract

Learning the necessary high-level reasoning for video stabilization without
the help of optical flow has proved to be one of the most challenging tasks in
the field of computer vision. In this work, we present an iterative frame
interpolation strategy to generate a novel dataset that is diverse enough to
formulate video stabilization as a supervised learning problem unassisted by
optical flow. A major benefit of treating video stabilization as a pure RGB
based generative task over the conventional optical flow assisted approaches
is the preservation of content and resolution, which is usually obstructed in
the latter approaches. To do so, we provide a new video stabilization dataset
and train an efficient network that can produce competitive stabilization
results in a fraction of the time taken to do the same with the recent
iterative frame interpolation schema. Our method provides qualitatively and
quantitatively better results than those generated through state-of-the-art
video stabilization methods. To the best of our knowledge, this is the only
work that demonstrates the importance of perspective in formulating video
stabilization as a deep learning problem instead of replacing it with an
inter-frame motion measure.

##  1 Introduction

The advent in the field of visual sensory has enabled people on a budget to
record high-resolution videos through their mobile phones, but it requires
additional hardware-assisted equipment to obtain stable videos (e.g., optical
image stabilizer). Although the quality of these videos can be enhanced to an
extent using digital video editing software, it requires additional time and
resources to stabilize the shaky videos. Thus, considerable literature has
been studied to solve the video stabilization problem, and enforcing
temporally smooth camera motion leads to the success of the stabilization
tasks [6, 15]. However, these approaches have difficulties in dealing with
objects with distinct motion profiles. To mitigate this problem, feature
trajectories are regularized rather than the global camera trajectory with the
assistance of dense optical flow [14]. Although quite successful, these flow-
based approaches are prone to loss of quality and visual information as their
dependence on optical flow makes them vulnerable where correspondences are not
well established leading to a large amount of cropping at the frame boundaries
when the motion displacement is significantly large.

Figure 1: (a) A visual representation of perspective mismatch and large non-
overlapping regions (transparent red zone) present in the DeepStab dataset
[23] along with the minimized perspective difference in our dataset. The
presented sub figures are unstable and stable versions of DeepStab dataset
[23] and ours from top to bottom, (highlighted with red, blue and green colors
respectively). (b) An illustration of how the perspective of stable and
unstable counterparts in the DeepStab dataset can diverge with jerky motions
and how our proposed dataset generation method compensates the effect while
only eliminating the high frequency jitters.

Recently, Wang et al. [23] released the DeepStab dataset, which is the first
large dataset for video stabilization, and there have been several attempts to
solve the stabilization problem using deep neural networks since then.
Specifically, the DeepStab dataset is captured with two synchronized cameras
from two different perspectives, and only one of them is stabilized by a
hardware stabilizer (e.g., DJI Osmo).

However, two frames captured at the same time from different viewpoints can
include a large non-overlapping field-of-view when there is a sudden
rotational motion even with a small disparity between the cameras as shown in
fig:fig1 (a)-(b). The frames presented with red and blue borders are the
original unstable and stable frames from the DeepStab dataset [23], whereas
the frames highlighted in green are produced by our dataset generation
pipeline. Due to this variation of perspective, recent neural networks trained
with the DeepStab dataset [23] cannot directly predict the stabilized video
frames, and inevitably employ intermediate steps to warp and crop the frames
using optical flow.

Based on these observations, we present a pipeline to generate a new video
stabilization dataset capable of formulating video stabilization as an image-
based supervised learning task consisting of pairs of stable and unstable
video frames with a minimized perspective difference. More specifically, we
extend and improve the idea of iterative frame interpolation leading to
smoothened motion trajectories presented in [2]. In general, video frame
interpolation methods produce a middle frame given its neighboring frames,
this generation of the middle frame can remove high-frequency jitters present
between the alternate frames. When used recursively, these methods can take
into account the relative motion of all the frames present in the sequence and
generate a temporally smooth sequence free from sudden high-frequency jerks as
shown in fig:fig2. To generate the dataset, we experiment with state-of-the-
art video frame interpolation techniques such as [3, 20]. Both of these video
frame interpolation methods can generate stable frames but are prone to image
degradation in iterative environments. In order to account for the artifacts
produced by these methods, we introduce a video refinement network that can
significantly improve the quality of the generated frames given the original
unstable (relatively sharper) and stable but degraded frames.

Using the generated large-scale video stabilization dataset, which includes a
much smaller perspective difference between the stable and unstable
counterparts, we can train convolutional neural networks (CNNs) to regress the
latent stabilized frames from the given input shaky video frames directly
without relying on the assistance of optical flow. Moreover, our proposed
network utilizes an adversarial training pipeline to produce photo-realistic
videos and provides state-of-the-art video stabilization results on publicly
available benchmark datasets. We summarize our contributions as follows:

  * •

Dataset Generation: We Introduce a novel frame interpolation based strategy to
produce equi-perspective stabilized videos from any unstable videos captured
from hand-held devices.

  * •

Fast stabilization network: We propose a fast RGB based regressor that does
not require any explicit flow information for stabilizing videos.

##  2 Related Work

Most of the work done in this field can be classified into three distinct
domains: 3D transformation, 2D transformation, and optical flow assisted
methods.

Liu et al. [12] proposed a 3D approach for this task, in which the positions
of the camera along with feature tracks were reconstructed in a 3D space, and
the feature positions were projected along smoothened camera positions,
whereas, Smith et al. [22] employed depth-aware cameras to do the same.

Figure 2: A visual description of iterative frame interpolation leading to
visual stability and smoothened camera trajectory.

However, these 3D approaches cannot handle dynamic scenes including moving
objects, and 2D transformations are the most common methods used in video
stabilization applications. In general, these methods rely on tracking
prominent features and stabilizing their trajectories along the motion path.
The results produced by these methods are generally up-scaled and cropped
around the borders to retain the original resolution of the input video.
Buehler et al. [1] estimated the camera positions through shaky videos and
rendered the frames at smoothened camera positions using a non-metric Image
Based-Rendering method. Matsushita et al. [18] and Gleicher et al. [4] used a
simplistic 2D transformation mechanism to warp the original frames. Whereas,
Liu et al. [15] introduced a grid-based warping of frames for smoothing the
feature trajectories. Grundmann et al. [6] proposed an L1 based method for
obtaining optimal camera trajectory for stabilized feature tracks, whereas Liu
et al. [13] proposed a similar approach but employed eigen-trajectory
smoothing. Wang et al. [24] and Goldstein et al. [5] also approached this task
with a constraint-based model to acquire feature tracks and position in
cartesian and epipolar geometry, respectively.

All of these methods relied heavily on feature tracks and ignored the
underlying relation of independent motion of objects in consecutive frames,
which compelled Liu et al. [14, 16] to investigate applications of optical
flow in the field of video stabilization. Their studies helped in
understanding the importance of inter-frame motion estimation in video
stabilization and paved the path for modern video stabilization methods. Yu
and Ramamoorthi [26, 27] and Choi et al. [2] employ dense optical flow
estimation modules to warp the neighboring frames to obtain smoother and
better-quality videos. In particular, Yu and Ramamoorthi [26] proposed a
scene-based deep learning network that estimates dense motion to directly
generate a warp field and extended their approach in [27] to handle complex
situations including (de)occlusion and non-linear motion.

Two pioneering implicit optical flow-based methods were proposed in the works
of Xu et al. [25] and Wang et al. [23]. These methods employed generative
adversarial networks and spatial transformer network to implicitly learn the
inter-frame motion and warping of the frames for video generation. Wang et al.
[23] attempted to find a possible solution with a Siamese network containing a
pre-trained ResNet50 model and created the only available dataset (DeepStab)
with pairs of synchronized stable and unstable frames. To create the dataset,
a hand-held mechanism employing two synchronous cameras, one of which was
fixed rigidly and the other one is stabilized by a gimbal stabilizer. The
rigidly fixed camera recorded the shaky videos, whereas, the camera placed on
the stabilizing equipment recorded the stabilized version of the same scene
from a slightly different perspective. This presence of perspective mismatch
made it challenging to learn the video stabilization when used in a
conventional supervised learning strategy and encouraged an un-supervised
approach by Choi et al. [2] to investigate and employ the video frame
interpolation method.

An attempt to train an image-based stabilizer using the DeepStab dataset
without motion estimation was discussed in [27] and was abandoned and termed
as an “essential over-fitting” task because using this dataset as an input and
ground truth pair can lead to the network learning an entirely different
perspective of the same scene without the presence of any correlation or
information about the perspective in the input unstable video frame when there
is a rotational motion. Whereas, the attempt to train the a unsupervised flow
assisted stabilizer using video interpolation by [2] was successful in
learning meaningful spatial relations. These attempts ([2, 27]) helped us
understand the importance of an often neglected factor (perspective) in the
dataset for the video stabilization task and encouraged us to formulate video
stabilization as a supervised learning task unassisted of optical flow
information.

Figure 3: Motion compensation comparison between the quality of network
trained on synthetic dataset (a) DIFRINT [2] and the network trained on real
world scenarios (b) CAIN [3]. The results presented above are acquired by
following the default procedure (i.e. 3 iterations with 2 skipped frames)
defined in [2] for stabilization. Figure 4: Visual artifacts comparison when
$m=20$. (a) Results with SepConv [20]. (b) Results with CAIN [3]. (c) Refined
CAIN results. Figure 5: (a) Training scheme for the refinement network. (Step
1) Generation of high-quality training data for the refinement network by
linearly cropping high-quality frames. (Step 2) Introduction of the
conventional frame-interpolation corruption through iterative schema. (Step 3)
Training pipeline of the employed Refinement Network. This network takes in
sequential high-quality frames $(c_{t-2},c_{t-1},C_{t+1},c_{t+2})$ produced
during step 1 along with a single degraded interpolated frame $i_{t}$. Note
that $C_{t}$ is clean ground truth patch of $i_{t}$. (b) Our dataset
generation pipeline for video stabilization. We integrate the Refinement
Network inside the iterative frame interpolation arrangement. The refinement
network takes in original unstable but shapr frames
$(U_{t-2},U_{t-1},U_{t+1},U_{t+2})$ and one interpolated frame $I_{t}$ to
render a clean frame.

We first create a new dataset containing stable counterparts of unstable
videos provided in the DeepStab dataset [23] along with the videos acquired
through online video sharing platforms like YouTube with a similar
perspective, aided by the state-of-the-art frame interpolation method [3]
(discussed in Sec. 3), and then develop a fast video stabilization network
which can produce a high-quality stable video from a shaky input video
(described in detail in Sec. 4).

##  3 Dataset Generation

The generation of a labeled dataset for video stabilization is a challenging
task. Attempting this task with hardware-assisted equipment like the one used
by Wang et al. [23] leads to the capture of the same scene from mismatched
perspectives with a significant non-overlapping region since the hardware-
assisted stabilizer is too insensible to the motion change as illustrated in
fig:fig1. This inconsistency in the perspective of stable and unstable
counterparts of the acquired dataset complicates the stabilization pipeline
and sway the researchers [2, 27] to look for other methods or datasets for
generating the training data.

Choi et al. [2] proposed a frame-interpolation-based network. Specifically,
the first part of their network is similar to the conventional frame
interpolation networks, which is trained with an affine translation-based
synthetic dataset. This synthetic dataset lacks the complexity of real motion
and only takes into account the global motion profiles. Thus, this network
faces difficulties in handling the differently moving objects in real-world
scenarios and can result in undesirable wobble like artifacts as shown in
fig:fig3 (a). In contrast, in the modern frame interpolation methods [3, 20],
this is compensated by training the networks on real-world videos containing
complex inter-frame local and global motions as shown in fig:fig3 (b). After
extensive experimentation to generate a meaningful dataset, we extended and
improved the iterative stabilization approach presented in [2] with a state-
of-the-art video frame interpolation method [3] to utilize the information
contained in the entire sequence without skipping the intermediary frames and
reconstructed the stable versions of the unstable videos captured with
ordinary hand-held cameras.

Before finalizing the baseline network for dataset generation, we tested two
state-of-the-art frame interpolation methods [20] and [3] in an iterative
arrangement where the output of the network in one iteration is fed again to
the same network. We repeat the process for a fixed number of iterations _m_
to achieve temporal stability. When used iteratively, both of these methods
provided similar stability, but the visual quality of the produced frames was
quite different as shown in fig:fig4 (a) and (b). Notably, CAIN [3] produced
lesser artifacts and relatively better quality frames, making it the best
choice for the task of data generation in our study.

Figure 6: (a) Network architecture of the proposed fast stabilization network
along with the discriminator for training. (b) Training strategy: stage 1 for
stabilization (left) and stage 2 for quality enhancement (right).

##### Dataset Refinement

To remove the produced artifacts by the iterative frame interpolation
arrangement, a refinement network is proposed. This network merges and takes
advantage of the visual quality improvement techniques and modules proposed in
[3, 11, 17, 19, 21] for their proven success in generating visually pleasing
images. As shown in fig:fig5 (a), the training data for this network is
generated by a linearly moving window on high-quality images. These cropped
images are passed through the iterative frame-interpolation pipeline to create
degraded frames containing artifacts similar to the ones. It is worth
mentioning that the training data generated for this network contains pairs of
flawed and high-quality frames with absolute overlapping and no perspective
difference unlike the training data used for the stabilization network. A
representation of the refinement pipeline is provided in fig:fig5 (b). Please
refer to the supplementary material for the detailed configuration and
training strategy for the refinement network. Through empirical analysis, it
was determined that the best results were produced by integrating the
refinement network within the iterative arrangement of the frame interpolation
pipeline. During the generation of the training dataset for the stabilization
network, we introduced a refinement step after every _k_ iterations of the
frame interpolation network and repeat this setting for a definite number _m_
of times to acquire temporally stable and high-quality frames as shown in
fig:fig4 (c). In our experiments, _k_ and _m_ were fixed to 4 and 5,
respectively.

##  4 Proposed Method

###  4.1 Fast Stabilization Network

The conventional video stabilization methodologies rely heavily on the
calculation of dense optical flow. The acquisition of dense optical flow is a
computationally expensive task, and various techniques are required to enforce
the temporal consistency of the acquired flow maps for further processing as
described in [27]. Although the proposed stabilization mechanism used to
generate our dataset through the iterative frame interpolation framework does
not depend on optical flow, it lacks the robustness to predict sequential
stable frames, and requires significant computing resources and time. The
laborious nature of these pipelines and the added complexity of optical flow
processing make these frameworks difficult to be employed in real-world
applications; therefore, we propose a neural approach that can efficiently
learn to produce promising results to the pipelines mentioned above by
removing the reliance on the optical flow. Various model architectures like
U-Net and ResNet based structures were tested before finalizing the baseline
architecture for this task. The baseline architecture used in this process is
a slightly modified version of ENet [21] based on ResNet architecture proposed
for super-resolution.

Having a very deep ResNet based structure (i.e., ENet [21]) as the backbone of
the stabilization network taking multiple consecutive frames as input allows
the network to exploit spatio-temporal information along with an extended
receptive field. The architecture of the proposed network for video
stabilization is shown in fig:fig6. Our model takes in five neighboring frames
and produces the stabilized version of the middle frame, these five frames are
concatenated along the channel dimensions before passing through the network.
Having multiple unstable frames as inputs allow the model to utilize a larger
set of related information to generate a spatially consistent and wobble free
frame as described in [23]. The number of input frames used for stabilization
is determined empirically. Instead of using any explicit module for optical
flow estimation, the proposed network learns to compensate motion implicitly
with the added depth of 64 residual blocks.

###  4.2 Training Strategy

In this section, we describe the details of the training strategy along with
the employed loss functions. We divide the training into two distinct stages,
namely, _Stabilization stage_ and the _Quality improvement stage_.

####  4.2.1 Stage 1: Stable Frame Generation

In this stage we train the network with a specific goal of generating stable
frame _$f^{\prime}_{t}$_ from the five unstable input frames
$(f_{t-2},f_{t-1},f_{t},f_{t+1,},f_{t+2})$. During this stage, the perceptual
quality of the generated frames is ignored purposefully as it can be enhanced
in the next stage with the help of an adversarial training strategy and a
perceptual loss. With this strategy, the model can focus on learning the high-
level reasoning necessary to justify the generated output $f^{\prime}_{t}$
from the input unstable frames. In this stage, we train the stabilization
network with the $\mathcal{L}_{2}$ based reconstruction loss as

$\mathcal{L}_{2}={\left||{f}^{\prime}_{t}-{f}_{gt,t}\right||}_{2}^{2},$ (1)

where _f gt,t_ is the stable counterpart of the output frame
_$f^{\prime}_{t}$_ , acquired through the iterative dataset generation
pipeline (described in Sec. 3). This loss was chosen empirically as we also
experimented with $\mathcal{L}$1 loss which resulted in relatively sharper
images, but it increased the convergence time. Since the second stage is
entirely dedicated to quality enhancement, this slight improvement in the
quality can be ignored at this stage for relatively faster convergence. For
both stages of training, we use ADAM for minimization, and for the first
frame, we initialize the learning rate as 0.0001 and use a plateau-based
decreasing strategy for the learning rate to facilitate the optimization
process.

####  4.2.2 Stage 2: Quality Enhancement

After the convergence with the $\mathcal{L}_{2}$ reconstruction loss in (1),
the results produced by the network are stable but quite blurry. By overlaying
the generated frames with the target frames, it becomes evident that the model
has learned the necessary high-level reasoning to generate stable frames, as
illustrated in fig:fig7 (a) and (b), where the red marks indicate the
positions of prominent corners in Refined CAIN and generated (Stage 1)
results. The perceptual quality of these blurry frames can be further improved
by fine-tuning the network with a perceptual and an adversarial loss for their
proven success in enhancing the visual quality of degraded images [8, 10, 19].
The primary loss used during this stage is a VGG based loss defined as follows

$\mathcal{L}={\left||\phi\left({f}^{\prime}_{t}\right)-\phi\left({f}_{gt,t}\right)\right||}_{2}^{2},$
(2)

where _$\phi\left({f}^{\prime}_{t}\right)$_ are the VGG features extracted
from the generated frame _$f^{\prime}_{t}$_ , and
_$\phi\left({f}_{gt,t}\right)$_ are those extracted from ground-truth target
frame _f gt,t_. For the purpose of feature extraction, a pre-trained VGG-19 is
used up to the _relu $\textunderscore$3$\textunderscore$3_ layer. This loss
ensures the preservation of high-level visual cues present beyond the single
pixel domains as proposed in [7].

The discriminator used in our application is shown in fig:fig6 (a). It is a
simple feed-forward network inspired by the discriminator used in [9] with
alternating convolution and Leaky-Relu operations.

The final loss for training in this stage is given by the following equation

$\mathcal{L}=\mathcal{L\textsubscript{content}}+\lambda\cdot\mathcal{L\textsubscript{adv}},$
(3)

where $\mathcal{L}_{content}$ represents the perceptual loss,
$\mathcal{L}_{adv}$ is the adversarial loss and $\lambda$ denotes a user-
parameter that controls the contribution of the adversarial loss in the
optimization step. The value of lambda is set to 0.01 throughout the second
stage of training. To ensure the retention of already learned relations during
the first stage, a considerably small learning rate of $5{\times}{10^{-5}}$
with a standard linear decay policy provided by PyTorch, is used during the
second stage. The results produced after the second stage of training contain
sharp edges while maintaining the spatial relations learned during stage 1, as
shown in fig:fig7 (c).

Figure 7: The results generated by our network after the both training stages.
(a) The red indicators represent the same position on Refined CAIN and the
Stage 1 results. (b) Highlight the alignment by overlaying the generated
frames with the frames acquired by the refined CAIN after the first training
stage. (c) Quality improvement after the second stage.

##  5 Experiments

This section summarizes the results produced by the extensive comparison of
the proposed model and state-of-the-art methods available for video
stabilization. The visual quality of the results generated by our model can
also be assessed through the supplementary video.

###  5.1 Implementation Details

The proposed stabilization network in Sec. 4 is implemented using PyTorch on a
system with two 2080Ti GPUs. We employ various augmentation techniques like
random flipping both horizontally and vertically, resizing and reversing the
frame order along with the conventional image processing augments like random
brightness, hue, gamma, and contrast adjustment are introduced to increase and
modify the visual information present in the training samples. We employ a
random patch-based training strategy as used in [3]. The patch size used
throughout the training process is 220$\times$220 for both stages, meanwhile,
the batch size used during the first stage is 5 and for the second stage it is
reduced to 3, and there are 5 patches per batch. The complete training process
(both stages) takes around three days with roughly 70,000 training iterations
in each step.

###  5.2 Quantitative Results

First, we evaluate the performance of the proposed method quantitatively in
terms of metrics (i.e., stability, distortion, cropping) as suggested by [15]
on the 6 categories presented in the NUS dataset [15]. The provided results
are averaged over each category. The details of these metrics are provided
below.

Figure 8: Robust L1 [6], DIFRINT [3], CAIN ($m=20$) [3] and our stabilization
network results are quantitatively compared in terms of stability, distortion,
and cropping.

##### Stability

This metric defines the stability in terms of frequency component analysis. To
calculate this metric, the feature trajectories are analyzed in the frequency
domain as follows

$f\textsubscript{x}=FFT(V\textsubscript{x}),$ (4)

where _f x_ is frequency representation of the translational and rotational
camera trajectories _V x_. _f x_ is acquired through the discrete 1D Fourier
Transform of _V tr,x_ after subtracting the DC component. Here, x represents
both translational and rotational trajectories. The stability score is
calculated according to the description provided by [23] as follows:

$S\textsubscript{x}=\sum\limits_{k=2}^{6}f\textsubscript{x}(k)/\sum\limits_{k=2}^{n}f\textsubscript{x}(k),$
(5)

where _S x_ represents the stability score for both translational and
rotational motions and _n_ is the number of total frequency components present
in the signal. The final stability score from both the calculated scores is
acquired by taking the minimum as:

$S\textsubscript{final}=min(S\textsubscript{t},S\textsubscript{$\theta$}).$
(6)

Figure 9: Visual quality comparison of input frames against Adobe Premiere
2018 CC, Robust L1 [6], DIFRINT [2], CAIN [3] (20-iterations), Refined CAIN,
and Our network output.

##### Distortion

This metric evaluates the anisotropic homography of the generated frames to
the actual unstable frames. The lowest ratio is selected as the final
distortion score. A higher score in this metric signifies better preservation
of the content.

##### Cropping

This metric measures the retention of visual information in generated frames
through homography calculation between the generated and the actual frames. A
higher score signifies better preservation of the visual information.

fig:quan_result (a) compares the average stability scores of Grundmann et al.
[6], DIFRINT [2], CAIN [3] (20 iterations) and the results produced by the
proposed fast stabilization network for all the 6 categories present in the
NUS dataset. Whereas, fig:quan_result (b) and (c) compare the distortion and
cropping scores for the same. Our network produces comparative results to
state-of-the-art video stabilization methods while preserving the temporal
consistency and finer details that are neglected in the other methods in less
than half the time taken by the other methods. Our method performs slightly
less in the regular category as the dataset generated for training did not
include any scene with a stationary camera, this can be avoided by fine-tuning
the model with sample scenes containing similar scenarios.

##### Run-time

The average per frame time comparison of all the methods presented above is
provided in Tab. 1.

Table 1. Time comparison |  |
---|---|---
DIFRINT [2] | 528ms |  |
Grundmann et al. [6] | 600ms |  |
CAIN (20-iterations) [3] | 1802ms |  |
Refined CAIN (20-iterations) | 2300ms |  |
Ours | 210ms |  |

Through Tab. 1 it can be deduced that our proposed model produces competitive
results in lesser time. All the experiments are conducted on the same hardware
and software environment as described in Sec. 5.1.

###  5.3 Qualitative Results

For visual quality comparison, we compare the results generated by Adobe
Premiere 2018 CC, Grundmann et al. [6] (Robust L1), Choi et al. [2] (DIFRINT),
Liu et al. [15] (Bundled), Choi et al. [3] (CAIN), frames generated through
the dataset generaton pipeline (Refined CAIN) and the output from the proposed
stabilization model in fig:Quality.

The loss of visual resolution can be clearly seen in the Adobe premiere 2018
CC. L1 compensates for the loss by upscaling the output frames. Meanwhile, the
bounded yellow regions in DIFRINT [2] and CAIN [3] highlight the artifacts
generated by both methods. It can be seen from these results that our model
produces better visual results along with preserving the scale and content.
More results can be seen in the provided supplementary material.

##  6 Discussion

The biggest benefit of supervised learning approaches with deep learning
models is the direct proportionality of predictive accuracy to the diversity
of the dataset. Lack of this diversity can sometimes lead to unexpected
results in unconstrained environments. This can easily be avoided by fine-
tuning the network on a different dataset generated through the dataset
generation pipeline containing similar scenarios as discussed earlier in Sec.
3. A simpler yet effective solution to the inconsistencies produced can be the
use of another module that can be trained on a dataset consisting of stable
videos with similar scenarios passed through the stabilizing network with the
original stable videos as the targets. A minor limitation of frame
interpolation based methods is their treatment of non-linear motion in low fps
videos. Sudden non-linear motion within the consecutive frames can lead to
inconsistent results containing linear motion and artifacts.

##  7 Conclusion

In this paper, we present a practical full frame supervised learning based
videos stabilization solution by demonstrating the importance of perspective,
which is often neglected in modern video stabilization approaches. We also
propose an iterative frame interpolation based solution for creating an
extensive datasets consisting of scenes from unconstrained environments to
further the investigation of generative solutions for this task instead of the
conventional spatial transformation approaches. Through our extensive
experimentation we have evaluated that the proposed network compares favorably
to the state-of-the-art solutions in terms of stability and performs
exceptionally well in preserving the visual information as well as the
resolution.

A future direction of this study can be the transformation of this offline
method to a causal online method by utilizing the immediate historically
stabilized frames instead of relying on past and future frames.

## References

  * [1] Chris Buehler, Michael Bosse, and Leonard McMillan.  Non-metric image-based rendering for video stabilization.  In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, pages II–II. IEEE, 2001.
  * [2] Jinsoo Choi and In So Kweon.  Deep iterative frame interpolation for full-frame video stabilization.  ACM Transactions on Graphics (TOG), 39(1):1–9, 2020.
  * [3] Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee.  Channel attention is all you need for video frame interpolation.  In AAAI, pages 10663–10671, 2020.
  * [4] Michael L Gleicher and Feng Liu.  Re-cinematography: Improving the camerawork of casual video.  ACM transactions on multimedia computing, communications, and applications (TOMM), 5(1):1–28, 2008.
  * [5] Amit Goldstein and Raanan Fattal.  Video stabilization using epipolar geometry.  ACM Transactions on Graphics (TOG), 31(5):1–10, 2012.
  * [6] Matthias Grundmann, Vivek Kwatra, and Irfan Essa.  Auto-directed video stabilization with robust l1 optimal camera paths.  In CVPR 2011, pages 225–232. IEEE, 2011.
  * [7] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [8] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas.  Deblurgan: Blind motion deblurring using conditional adversarial networks.  ArXiv e-prints, 2017.
  * [9] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas.  Deblurgan: Blind motion deblurring using conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8183–8192, 2018.
  * [10] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang.  Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better.  In The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
  * [11] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [12] Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala.  Content-preserving warps for 3d video stabilization.  ACM Transactions on Graphics (TOG), 28(3):1–9, 2009.
  * [13] Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin, and Aseem Agarwala.  Subspace video stabilization.  ACM Transactions on Graphics (TOG), 30(1):1–10, 2011.
  * [14] Shuaicheng Liu, Ping Tan, Lu Yuan, Jian Sun, and Bing Zeng.  Meshflow: Minimum latency online video stabilization.  In European Conference on Computer Vision, pages 800–815. Springer, 2016.
  * [15] Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun.  Bundled camera paths for video stabilization.  ACM Transactions on Graphics (TOG), 32(4):1–10, 2013.
  * [16] Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun.  Steadyflow: Spatially smooth optical flow for video stabilization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4209–4216, 2014.
  * [17] Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou.  Structure-preserving super resolution with gradient guidance.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [18] Yasuyuki Matsushita, Eyal Ofek, Weina Ge, Xiaoou Tang, and Heung-Yeung Shum.  Full-frame video stabilization with motion inpainting.  IEEE Transactions on pattern analysis and Machine Intelligence, 28(7):1150–1163, 2006.
  * [19] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3883–3891, 2017.
  * [20] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In Proceedings of the IEEE International Conference on Computer Vision, pages 261–270, 2017.
  * [21] Mehdi SM Sajjadi, Bernhard Scholkopf, and Michael Hirsch.  Enhancenet: Single image super-resolution through automated texture synthesis.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4491–4500, 2017.
  * [22] Brandon M Smith, Li Zhang, Hailin Jin, and Aseem Agarwala.  Light field video stabilization.  In 2009 IEEE 12th international conference on computer vision, pages 341–348. IEEE, 2009.
  * [23] Miao Wang, Guo-Ye Yang, Jin-Kun Lin, Song-Hai Zhang, Ariel Shamir, Shao-Ping Lu, and Shi-Min Hu.  Deep online video stabilization with multi-grid warping transformation learning.  IEEE Transactions on Image Processing, 28(5):2283–2292, 2018.
  * [24] Yu-Shuen Wang, Feng Liu, Pu-Sheng Hsu, and Tong-Yee Lee.  Spatially and temporally optimized video stabilization.  IEEE transactions on visualization and computer graphics, 19(8):1354–1361, 2013.
  * [25] Sen-Zhe Xu, Jun Hu, Miao Wang, Tai-Jiang Mu, and Shi-Min Hu.  Deep video stabilization using adversarial networks.  In Computer Graphics Forum, volume 37, pages 267–276. Wiley Online Library, 2018.
  * [26] Jiyang Yu and Ravi Ramamoorthi.  Robust video stabilization by optimization in cnn weight space.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3800–3808, 2019.
  * [27] Jiyang Yu and Ravi Ramamoorthi.  Learning video stabilization using optical flow.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8159–8167, 2020.