# Frame Difference-Based Temporal Loss for Video Stylization

Jianjin Xu, Zheyang Xiong, Xiaolin Hu  J. Xu and X. Hu are with the Department
of Computer Science and Technology, Tsinghua University, Beijing 100084,
China. J. Xu is also with the Department of Computer Science, Columbia
University, New York 10027, NY, USA.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Z. Xiong is with the Department
of Computer Science, Rice University, Houston 77005, TX, USA.
E-mail<EMAIL_ADDRESS>
###### Abstract

Neural style transfer models have been used to stylize an ordinary video to
specific styles. To ensure temporal inconsistency between the frames of the
stylized video, a common approach is to estimate the optic flow of the pixels
in the original video and make the generated pixels match the estimated
optical flow. This is achieved by minimizing an optical flow-based (OFB) loss
during model training. However, optical flow estimation is itself a
challenging task, particularly in complex scenes. In addition, it incurs a
high computational cost. We propose a much simpler temporal loss called the
frame difference-based (FDB) loss to solve the temporal inconsistency problem.
It is defined as the distance between the difference between the stylized
frames and the difference between the original frames. The differences between
the two frames are measured in both the pixel space and the feature space
specified by the convolutional neural networks. A set of human behavior
experiments involving 62 subjects with 25,600 votes showed that the
performance of the proposed FDB loss matched that of the OFB loss. The
performance was measured by subjective evaluation of stability and stylization
quality of the generated videos on two typical video stylization models. The
results suggest that the proposed FDB loss is a strong alternative to the
commonly used OFB loss for video stylization.

###### Index Terms:

Video stylization, stability, temporal consistency, frame difference

##  1 Introduction

In recent years, neural style transfer for images has achieved great success.
Given a style image (e.g., “The Starry Night” by Vincent van Gogh), these
models can transfer an ordinary image to an image with the same style as the
style image, which is called stylized image. The first method reported in this
field was an optimization-based algorithm [1] that adjusts the input
iteratively according to certain goals defined by convolutional neural
networks. Subsequently, many computationally more efficient learning-based
neural networks were proposed[2, 3, 4, 5, 6, 7].

An interesting extension of these models is to stylize videos. Specifically,
given a style image and a video, a new video is generated for which the frames
have a similar style to the style image. Direct generation of video frames
using these models will lead to high-frequency flickering artifacts. This
deficiency is called temporal incoherence.

To solve the temporal incoherence problem, most video stylization models track
the motion of pixels along time in the original video and use this information
to stylize videos. This leads to the need for optical flow [8]. Many
algorithms have been proposed to estimate optical flow in videos, including
(slow) iterative algorithms [8, 9] and (fast) deep learning models [10, 11].
Most algorithms are developed under the assumption of brightness constancy—a
pixel’s intensity or color does not change when it flows from one frame to
another [9]. This assumption alone leads to an ill-posed problem (one
constraint for two unknowns at each pixel), and different algorithms use
different prior terms for regularization (see [9] for a review).

It is known that many issues remain unsolved for optical flow estimation in a
scene due to effects such as ambient disturbance, illumination change, and
occlusion. A perfect algorithm that can solve all of these problems does not
exist, and errors are inevitable in certain cases. For example, it is
difficult for optical flow algorithms to predict the pixels’ movements near
the motion boundary due to occlusion. Fig. 1 shows a person and a horse in the
foreground moving to the right (Supplementary Video 1). Since the camera is
moving along with the person and the horse, the foreground is roughly static,
but the background is moving to the left. The bottom row shows the optical
flow results obtained using DeepFlow [12]. The black regions in the bottom-
middle indicate that the estimation near the boundary of the person and the
horse has very low confidence. This is because these regions in the background
are occluded by the person and the horse in the next frame, and the pixels in
these regions do not have corresponding pixels in the next frame. Clearly,
this situation worsens if the speed of motion is higher or the frame rate
(frames per second) is lower.

Figure 1: Inaccurate optic flow estimation due to occlusion. Top: two
successive frames in a video. Bottom: Left, the estimated forward optic flow.
Different colors indicate different motions, as illustrated on the right.
Middle, the occlusion mask of the optic flow. The value at each pixel is
either 0 (black) for the occluded area or 1 (white) for the area that can find
a corresponding pixel.

Can optical flow be replaced by other methods to stabilize videos during
stylization? We reconsider the nature of temporal coherence in videos. Since
the original video is assumed to be temporally coherent, the pixel values
change in a temporally coherent manner. If the pixel values in the stylized
video change in the same manner, then the stylized video should be temporally
coherent. We note that the frame difference uniquely determines the temporal
evolution of the pixel values in the original video. To achieve temporal
consistency, the difference between two consecutive frames in the generated
video can be made similar to the difference between the corresponding two
consecutive frames in the original video. A loss function can be designed in
this manner and combined with the original loss functions of any image
stylization model to produce temporally coherent videos.

Empirically, we found that this simple method produced excellent stability
results but also led to the degradation of stylization quality. We then
introduced the frame difference in the feature space specified by a neural
network and combined it with the frame difference in the pixel space as the
loss function. This loss is called frame difference-based (FDB) loss. It
produced stylized videos with a similar degree of video stability and
stylization quality to the optical flow-based (OFB) loss. Compared with the
OFB loss, the proposed FDB loss can significantly reduce the computational
cost because it does not estimate optic flow for the training videos before
training which typically requires a few days for a normal dataset.

The rest of the paper is organized as follows. In Section 2, some closely
related works are discussed. In Section 3, the FDB loss is formulated and
applied to two typical video stylization models. Section 4 presents the
experimental results. Section 5 concludes the paper.

##  2 Related Works

A comprehensive review of image stylization and video stylization is beyond
the scope of this paper. We only review some neural network-based methods that
are closely related to our method.

###  2.1 Neural image stylization

In 2015, Gatys et al. [1] proposed a novel method to transfer an image (called
content image) to a new image (called stylized image) with a style of another
image (called style image). This method is based on the assumption that a
pretrained deep neural network for classification on a large dataset can
represent the content and style of any image. The basic idea is to fix the
neural network and optimize the input such that its content representation and
style representation by the neural network match those of the content image
and the style image, respectively. However, optimization is an iterative and
time-consuming process. Learning-based methods have been proposed to directly
map the content image to the stylized image using a feedforward neural network
[2, 3], which are several orders of magnitude faster than the optimization-
based method. A different image stylization method is presented in [13] to
stylize an image according to the style defined by a set of images. One
limitation of these methods is that different models must be trained for
different style images. Several recent methods [4, 5, 6, 7, 14] have been
shown to be able to learn multiple styles simultaneously.

###  2.2 Neural video stylization

The critical problem for extending image stylization methods to stylize videos
is to ensure temporal coherence in stylized frames. By introducing a short-
term temporal consistency loss based on the optical flow in the frames, it was
shown that the optimization-based image stylization model [1] can produce
stable stylized videos [15]. However, these methods inherit the low speed of
the optimization-based image stylization method. It is preferred to start from
learning-based methods for image stylization [2, 3, 4, 5, 6, 7]. A successful
attempt of this was the model developed by Huang et al. [16] that uses the
image stylization model of Johnson et al.[2] but introduces a short-term
temporal loss for training the generative network. Another successful attempt
was carried out by Ruder et al.’s [17] who developed a model that also uses
the image stylization model of Johnson et al.[2] but introduces both short-
term temporal loss and long-term temporal loss. A recurrent neural network was
proposed to stabilize the generated video frames [18]. This approach differs
from that of the model of Huang et al. [16] in the generation process, in
which the output of the model at the previous time step is needed as input to
the model to generate the output at the current time step. With this setting,
the original video frames are required to be input to the model sequentially
during training and testing. A patch-based image stylization method [19] is
extended to stylize videos [20]. Image parts are tracked in different frames
based on an optical flow method and inconsistent patches are synthesized.
Another method estimates a low-resolution optic flow in every training step
and minimizes a compound temporal loss to stabilize the videos [21].

To ensure temporal coherence among video frames, the above methods usually
employ OFB temporal losses to train the stylization networks. After training,
the stylization networks are different from those trained without any temporal
loss. The stylized frames in the two settings are different; in other words,
the temporal loss may degrade the stylization quality of the individual
frames. Chen et al. [22] proposed to fix the stylization network but learn to
make use of the previous frame to avoid inconsistency. Each new frame is
obtained by linearly combining two frames with a learnable mask. Both frames
are the outputs of the stylization network, one from the current time step and
the other from the last time step but are warped to the current time step
using a feature level warp function based on optical flow. This method is
quite different from the existing methods and from the method proposed in this
paper because it does not change the stylization network.

###  2.3 Temporal consistency

In addition to video stylization, temporal consistency must be achieved in any
video editing application. Optical flow is also widely used in these
applications. Kong et al. [23] proposed to extract temporally coherent albedo
and shading from the video (intrinsic video) using optical flow. In addition,
it was shown that intrinsic video was useful to improve optical flow
estimation. Two different methods [24, 25] were proposed for intrinsic video
decomposition, where the temporal consistency was ensured by using optical
flow. Litwinowicz et al. [26] proposed to transform ordinary video into
animations that had an impressionist effect. Optical flow fields were used to
push brush strokes from frame to frame in a temporally consistent manner.
Farbman and Lischinski [27] proposed to stabilize tonal fluctuations in videos
by using optical flow estimated at sparse locations. Bonneel et al. [28]
proposed to stabilize many image processing tasks, including stylization of
color and tone and depth estimation using dense optical flow. Lai et al. [29]
proposed a cascaded solution to address the video stability problem in a wide
range of tasks by training an RNN using OFB temporal consistency loss. Most of
these methods consider the local temporal consistency between the adjacent
frames explicitly. A generative adversarial network-based method was proposed
to enhance the global consistency in many frames [30].

Since optical flow must be estimated in every frame, its estimation also faces
the stability problem. Many methods have been proposed for stabilizing the
optical flow across frames, including spatiotemporal filtering [31] and
temporal edge-aware filtering [32].

In addition to optical flow, several other techniques can also stabilize
frames in video editing. Motivated by Gaussian convolution, Paris [33] devised
an edge-preserving smoothing technique. Meka et al. [34] proposed a global,
sampling-based, and spatiotemporal reflectance consistency constraint. Bonneel
et al. [35] proposed a method based on high-dimensional differential geometry.
However, most of these methods are problem-specific and cannot be directly
used for neural video stylization.

##  3 Methods

We consider a class of video stylization models [16, 18] that uses a loss for
individual frames (called spatial loss in what follows) and a loss for
between-frames (called temporal loss). Our contribution is for a new temporal
loss, but to make the method self-contained, we briefly review the spatial
loss first.

###  3.1 Spatial loss

The spatial loss usually has two parts, content loss and style loss [1]. The
former is focused on preserving the content of the content image, and the
latter is focused on transferring the style to the style image. Current
learning-based neural networks for image style transfer usually rely on a
neural network called the loss network the parameters of which are pretrained
on other tasks (e.g., classification on the ImageNet dataset) and fixed. Both
content loss and style loss are computed on the top of this loss network and
are used to train a stylization network d parameters need to be learned.

We denote the content image by $I$, the style image by $S$ and the stylized
image (output) by $\tilde{I}$. Let $f_{l}(x)$ be the feature map of layer $l$
with $x$ being the input to the pretrained loss network, specifically, either
$I$ or $\tilde{I}$. For simplicity, batch size is set to 1. Then, a typical
neural network feature map has a 3D shape $C_{l}\times H_{l}\times W_{l}$,
where $C_{l}$ denotes the number of channels, and $H_{l}$ and $W_{l}$ denote
the height and width. The feature map $f_{l}(x)$ can be flattened into a 2D
matrix $F_{l}(x)$ with the shape $C_{l}\times H_{l}W_{l}$. The Gram matrix of
$f_{l}(x)$ is defined as [1]:

$G_{l}(x)=f_{l}(x)f_{l}(x)^{\top}.$

The content loss in a layer $l$ is defined as the Euclidean distance between
$I$ and $\tilde{I}$ at the feature level

$E_{\text{cont}}^{l}(\tilde{I},I)=\frac{1}{2C_{l}H_{l}W_{l}}\left\lVert
f_{l}(\tilde{I})-f_{l}(I)\right\rVert^{2}_{2}.$ (1)

The style loss in a layer $l$ is defined as the Euclidean distance between the
Gram matrices of the feature maps of $S$ and $\tilde{I}$

$E_{\text{sty}}^{l}(\tilde{I},S)=\frac{1}{2C_{l}^{2}}\left\lVert
G_{l}(\tilde{I})-G_{l}(S)\right\rVert^{2}_{2}.$ (2)

This loss is proposed in [1] and has been widely used in neural image
stylization [2, 3, 4].

The content losses and style losses from different layers are weighted and
summed together to yield the final losses

$L_{\text{cont}}(\tilde{I},I)=\sum_{l\in
J_{c}}a_{l}E_{\text{cont}}^{l}(\tilde{I},I)$ (3)
$L_{\text{sty}}(\tilde{I},S)=\sum_{l\in
J_{s}}b_{l}E_{\text{sty}}^{l}(\tilde{I},S)$ (4)

where $J_{c}$ and $J_{s}$ denote the sets of layers chosen to calculate the
content losses and style losses, respectively, and $a_{l}$ and $b_{l}$ denote
the weighting factors.

###  3.2 Temporal loss

Since the original video (or content video) is assumed to be temporally
coherent, and this coherence is encoded in the difference between frames, by
requiring the stylized video to have a similar between-frame difference in a
model, one may obtain stabilized video output.

We denote the original frames by $\\{I_{t}\\}$ and the stylized frames by
$\\{\tilde{I}_{t}\\}$. The temporal loss is defined as

$L_{\text{temp}}(\\{\tilde{I}_{t}\\},\\{I_{t}\\})=\frac{1}{2N(T-1)}\sum_{t=1}^{T-1}\left\lVert\phi(\tilde{I}_{t})-\phi(I_{t})\right\rVert_{2}^{2},$
(5)

where $N$ is the number of pixels in each frame, $T$ is the total number of
frames, and

$\phi(x_{t})=f_{l}(x_{t+1})-f_{l}(x_{t}),$ (6)

where $t=1,\ldots,T-1$, is the frame difference between two consecutive frames
$x_{t+1}$ and $x_{t}$. If $l=0$, $\phi(x_{t})$ is simply the frame difference
in the pixel space

$\phi(x_{t})=x_{t+1}-x_{t}.$ (7)

The temporal loss (5) with the frame difference defined in the pixel space
($l=0$) is called the pixel-FDB (P-FDB) loss, denoted by $L_{\text{pt}}$. If
$l>0$ in (6), then the frame difference is measured in the feature space, and
the loss is called the feature-FDB (F-FDB) loss and denoted by
$L_{\text{ft}}$. The P-FDB and F-FDB can be weighted and summed to form the
combined-FDB (C-FDB) loss, denoted by $L_{\text{ct}}$.

We would like to remark that measuring the difference between consecutive
frames is not the only choice for $\phi(x_{t})$. In fact, the difference
between any pair of frames $x_{t+K}$ and $x_{t}$ ($1\leq K\leq T-1$) can be
used, and their performances are similar (see Section 4.5 for empirical
results).

For any neural video stylization model, we can define the total loss for
minimization:

$L=\lambda_{1}L_{\text{cont}}+\lambda_{2}L_{\text{sty}}+\lambda_{3}L_{\text{temp}},$
(8)

where $\lambda_{i}$’s are the weighting factors. We note that
$L_{\text{temp}}$ can be $L_{\text{pt}}$, $L_{\text{ft}}$ or $L_{\text{ct}}$.

###  3.3 Application to video stylization models

(a) The SFN with the P-FDB loss.

(b) The SFN with the F-FDB loss.

(c) The RNN with the P-FDB loss.

(d) The RNN with the F-FDB loss.

Figure 2: Training pipelines for the SFN and the RNN. The P-FDB loss and the
F-FDB loss are defined in (6) and correspond to $l=0$ and $l>0$, respectively.
We note that in either case, the temporal loss needs four inputs:
$I_{t},I_{t+1},\tilde{I}_{t}$ and $\tilde{I}_{t+1}$. For the SFN, in
inference, at every time step $t$, only one frame $I_{t}$ from the original
video is used and a stylized frame $\tilde{I}_{t}$ is output. For the RNN, in
both training and inference, previous stylization output $\tilde{I}_{t}$ (or
zero at the first frame) and the current frame $I_{t+1}$ are used to produce
stylized output $\tilde{I}_{t+1}$.

Generally, the proposed FDB loss can be applied to any video stylization model
that uses the OFB loss. In this paper, we present its application to two
typical models:

  * •

A simple yet efficient model proposed in [16]. We refer to this as Simple
Feedforward Network (SFN).

  * •

A Recurrent Neural Network (RNN) proposed in [18].

Both models consist of a stylization network and a loss network. The
stylization networks are used to generate video frames and the loss networks
that are usually pretrained on the ImageNet classification task and then fixed
are used to calculate the losses for training the stylization networks. For
both models, we substitute the original OFB loss with the FDB loss and keep
other losses the same.

During training, the SFN takes in two successive frames to learn the temporal
coherence. The training pipelines with the P-FDB loss and the F-FDB loss are
slightly different, as illustrated in Figs. 2(a) and 2(b), respectively. We
note that the two frames form a minibatch with size 2 and are not concatenated
along the channel axis. Let the successive frames be $\\{I_{t},I_{t+1}\\}$ and
their stylized outputs be $\\{\tilde{I}_{t},\tilde{I}_{t+1}\\}$. Then, the
temporal loss in (5) is properly defined, and the total loss (8) is used to
optimize the stylization network. We note that there are two style losses for
each pair of inputs, two content losses, and just one temporal loss.

For comparison, we can replace the FDB temporal loss with the OFB temporal
loss to transform the SFN back to the model in [16]. A typical form of the OFB
loss is

$L_{\text{temp}}=\frac{1}{2N(T-1)}\sum_{t=1}^{T-1}\left\lVert
c_{k}(\tilde{I}_{t+1}-\omega(\tilde{I}_{t}))\right\rVert_{2}^{2},$ (9)

where $\omega$ is a function that warps the stylized output at time $t$ to
time $t+1$ according to the optical flow, and $c\in[0,1]^{N}$ denotes the per-
pixel confidence of the optical flow. Readers can refer to [15, 16, 18] for
the details of the OFB loss.

We note that during inference, the SFN takes one frame as input and outputs a
stylized frame, and the loss network is no longer needed.

The training pipelines for the RNN with the P-FDB loss and the F-FDB loss are
shown in Figs. 2(c) and 2(d), respectively. The RNN takes in two images
together, namely, the previous stylized frame $\tilde{I}_{t}$ and the current
content frame $I_{t+1}$. These are concatenated along the channel axis and
then fed into the RNN. Except for the difference in the input layer, the other
layers of the RNN are the same as the SFN. Since the “previous” stylization
frame of the first frame does not exist, it is set to a matrix of all zeros.
The temporal loss (5) is calculated based on $I_{t},I_{t+1},\tilde{I}_{t}$ and
$\tilde{I}_{t+1}$. In training and inference, the model works in the same
manner. Replacing the FDB loss with the OFB loss transforms the model back to
the model in [18].

##  4 Experiments

###  4.1 Dataset

The training split of DAVIS [36] was used as the training set. DAVIS is a
real-world video dataset covering several difficult scenarios such as fast
motion, motion blur, and occlusion. It consists of 50 video clips and
approximately 3,000 frames. No preprocessing was needed for the FDB loss,
while for the OFB loss, we estimated the optical flow for the entire dataset
prior to training. DeepFlow [12] was adopted for estimating the optic flow,
requiring the computational time of approximately 3 days.

Our test set consisted of randomly selected ten videos from the DAVIS test
split and ten videos from the Sintel [37] test split. Sintel is a computer-
generated movie dataset and is widely used in neural stylization studies.

###  4.2 Implementation details

We used the same architecture as [16] for the SFN but used a new padding
approach in every convolutional layer. We note that padding is necessary to
keep the size of the stylized image the same as that of the original image due
to the convolution operations in the model that shrink the size of the input
image. This network does not have pooling layers but has convolutional layers
with stride larger than 1. Even though upsampling layers are present, without
appropriate padding, the network still outputs smaller images than the input
images. In experiments, we found that the ordinary padding approaches were
prone to produce bounding box artifacts (horizontal or vertical bars appearing
around the boundary of the stylized video), even if no temporal loss was
applied. We found that this artifact could be largely suppressed by adopting
an interpolation padding approach. First, the feature maps with spatial size
$H\times W$ are bilinearly upsampled to the desired size $H^{\prime}\times
W^{\prime}$. Then, the interior parts with the spatial sizes of $H\times W$
are substituted by original feature maps. In other words, the padded area is
obtained by bilinear interpolation. This approach was adopted in both the SFN
and the RNN in all experiments. Its effect is demonstrated in Section 4.4.

The loss network was a modified version of VGG-16 [38] (all max-pooling layers
were replaced with average pooling layers as suggested in [1]), with its 16th
layer as the content layer and the 4th, 9th, 16th, 23rd layers as the style
layers. For comparison, we trained three SFN models with the P-FDB loss, the
OFB loss, and without any temporal loss, respectively. We tried our best to
make each model achieve its best performance by tuning the hyper-parameters.

The SFN with any temporal loss was trained in two steps. First, it was trained
as an image stylization model as described in [2]. We used the Adam [39]
optimizer with a learning rate of $10^{-3}$ and batch size 4 and optimized the
model for 20k iterations. Second, the SFN was finetuned on the DAVIS training
set with a certain temporal loss. The weights of the content loss and the
style loss were kept the same as in the first step to ensure that the
stylization did not change. The finetuning process required two epochs (3456
iterations) using the Adam optimizer with a learning rate of $10^{-4}$ and
batch size 2 (a pair of successive frames). We observed that two epochs of
training were sufficient to stabilize the videos. The first step took
approximately 6 hours. The training time for the second step depended on the
temporal loss but was always approximately 20 minutes. Without the temporal
loss, we only trained the SFN in the first step.

The RNN with any temporal loss was trained followed the setup in [18]. First,
the RNN was initialized with the SFN trained in the first step. The first
convolutional layer of the RNN that had six input channels was learned from
scratch. Second, the RNN was finetuned with a batch size of 4 according to
[18]. Four consecutive frames formed a minibatch. The other training settings
were the same as the SFN. The finetuning step took approximately one hour for
different temporal losses.

For all models, we set $\lambda_{1}=1$, $\lambda_{2}=10$ and
$\lambda_{3}=400$. For the C-FDB loss, we set
$L_{\text{ct}}=\frac{1}{40}L_{\text{pt}}+L_{\text{ft}}$. Therefore,
$\lambda_{3}L_{\text{ct}}=10L_{\text{pt}}+400L_{\text{ft}}$. The 9-th layer of
VGG16 ($l=9$ in (6)) was used to calculate the frame difference in the feature
space.

All of the experiments were conducted on a single NVIDIA GeForce GTX TITAN X
GPU. All models were implemented in Pytorch [40] and the source codes are
publicly available111https://github.com/AtlantixJJ/frame-difference-loss.

###  4.3 Evaluation method

We evaluate the video stability and frame stylization quality for models
trained with different temporal losses. It is difficult to find an objective
metric to measure perceptual stability and quality. Some widely used video
quality metrics, including PSNR and SSIM, do not accurately reflect human
perception. These metrics fail to account for many nuances of human perception
[41] that are important for the perception of the flicker artifacts in videos.
Therefore, we carried out behavioral experiments to evaluate the proposed FDB
loss. The study was approved by the Department of Psychology Ethics Committee,
Tsinghua University, Beijing, China.

We adopted the classical two-alternative forced choice (2AFC) experimental
paradigm. To evaluate video stability, pairs of videos were presented to the
subject simultaneously, and the subject was instructed to pick the more stable
video. Considering that a subject may not be familiar with what stability
means, we collected a few examples from demo videos (a stable video and an
unstable video) from previous video stylization studies [15, 22]. To evaluate
frame stylization quality, pairs of frames along with the corresponding style
image were presented to the subject, and the subject was instructed to select
the image based on the similarity of its style to the reference style image.
The contents of videos or frames and the style images were controlled to be
the same for comparing the performances of two different temporal losses.

To evaluate video stability, we stylized each test video into four videos with
style images, “Starry Night”, “La Muse”, “Feathers” and “Composition VII”
(Fig. 3). Since there were 20 videos in the test set, every method generated
80 videos for comparison. To evaluate frame stylization quality, we randomly
chose frames from the 80 stylized videos and ensured that the frame indices
were the same for different temporal losses to be compared. The order of
trials was randomly shuffled but fixed for all subjects.

Figure 3: Four style images used in this study.

For the comparison of a pair of losses (e.g., the P-FDB loss and the OFB
loss), one subject must finish two sets of evaluations: one set for video
stability and the other for frame stylization quality. This is called a
session. In other words, in every session, a subject should make 80 votes for
video stability and 80 votes for frame stylization quality. A subject must
complete all of the 160 votes before she or he can participate in another
session for comparing the results with another pair of losses. Each session
was assigned to 20 different subjects, resulting in $20\times 160=3,200$
votes. There were eight different sessions (three in Table I, three in Table
II and two in Table III). In total, we collected $8\times 20\times 160=25,600$
votes.

We recruited 62 subjects (31 male and 31 female, aged between 18 to 30). They
were paid according to the number of sessions completed. Among the subjects,
27 of them had normal vision. The others had corrected-to-normal vision.

We performed most analyses based on the SFN and present the results in
Sections 4.4 to 4.6). We present the main results of the RNN in Section 4.7.

###  4.4 Comparison of padding approaches

First, we found that if convolutions were used without padding, the bounding
box artifacts mentioned in Section 4.2 did not appear, but the stylized videos
had smaller spatial sizes than the original videos (Fig. 4, leftmost column).
As discussed in Section 4.2, to keep the spatial size of the original video,
padding is necessary. We tried different padding approaches (Fig. 4), namely:

  * •

Zero padding: put zeros for the surrounding of the feature maps;

  * •

Replicate padding: put the values of the border for the surrounding of the
feature maps;

  * •

Reflective padding: reflect every feature map to its surrounding as if there
was a mirror on each of the four borders;

  * •

Reflective-at-input padding222Used in implementing the image stylization
network in [2]. See https://github.com/jcjohnson/fast-neural-style: same as
reflective padding, but it is only used in the input image, while the
convolutional layers do not use any padding.

All of these approaches were prone to produce bounding box artifacts.
Specifically, for the zero padding approach and the reflective padding
approach, the artifacts appeared even without temporal loss. For the replicate
padding approach and the reflective-at-input padding approach, the artifacts
did not appear in the results of the SFN trained without the temporal loss but
appeared in the results of the SFN trained with the P-FDB loss. Instead, our
proposed interpolation padding approach as described in Section 4.2 did not
produce the bounding box artifacts.

Figure 4: Results of the SFN models using different padding approaches. The
first row illustrates various padding approaches. The second and the third
rows show the results of the SFN trained without temporal loss and with the
P-FDB loss, respectively. The style image is “La Muse”.

###  4.5 Results with the P-FDB loss

We first studied the FDB loss defined in the pixel space. The training
pipeline of the SFN is illustrated in Fig. 2(a).

####  4.5.1 Video stability: qualitative results

First, we compared the results of the SFNs trained with and without temporal
losses qualitatively. Without any temporal loss, the videos produced by the
SFN had flickering effects (see Supplementary Video 2 and Fig. 5). In row
“None” in Fig. 5, the zoom-in red box (upper part) demonstrates that the dark
blue textures on the bamboos in frame 1 change to sharp yellow stripes in
frame 2. In the blue box of the other pair of frames, the number and mode of
stripes of the two frames do not match333In our experiments, we found
difficult to demonstrate the instability of a video by comparing consecutive
frames, because the flickering effect usually had a high spatial frequency.
Readers are invited to watch the companion supplementary videos as indicated
in the captions of figures.. By contrast, both the P-FDB loss and the OFB loss
can stabilize the stylized videos. The bottom two rows in Fig. 5 show higher
consistency between two successive frames generated by the SFNs with the OFB
loss and the P-FDB loss.

Figure 5: Results of three SFNs given a content video (“bamboo_3” in the
Sintel dataset) and two style images (“Starry Night” and “La Muse”). The first
row shows the two style images and two successive frames in the content video.
Zoomed-in images are also shown as indicated by the red and blue boxes. The
second to the fourth rows show the stylized frames obtained by the SFNs
trained without temporal loss, with the OFB loss, and the P-FDB loss,
respectively. See Supplementary Video 2.

Second, we tested the approach of using a larger frame interval in the frame
difference measurement. As explained in Section 3.2, using the difference
between two frames that are separated by an interval of $K$ frames where $K>1$
should work as well as using $K=1$. This was verified by the experiments. As
shown in Fig. 6, the stylization results of models trained with different $K$
are almost identical to each other. See Supplementary Video 3.

Figure 6: Sample results of the SFNs with different frame intervals. The
models were trained with the style image “Feathers”. The first row shows three
original frames (No. 0, 1, and 9) from a clip in the DAVIS test split. The
second to the fourth rows show the stylized frames by SFN trained with the 1,
4, and 8 frame interval P-FDB losses, respectively. See Supplementary Video 3.

Third, we found that the models trained with the P-FDB loss were able to
maintain long-term consistency to some extent. An example is shown in
Supplementary Video 4 and Fig. 7. When the camera moved from left to right
beside a pillar in a scene, a part of the scene was occluded by the pillar for
some time and then reappeared. After reappearing, the regions before occlusion
were stylized in a consistent manner. By contrast, in the absence of the P-FDB
loss, the stripes in the red box before occlusion disappeared after this
region reappeared. We note that unlike a previous study [15], we did not
impose any explicit mechanism on the proposed model to ensure long-term
consistency. We speculate that this long-term consistency is a result of
learned resistance to the disturbance of input: if the input stays the same,
the stylized output is also trained to stay the same.

Figure 7: An example of long-term consistency in the models trained with the
P-FDB loss. The first row shows three frames (frame interval is 33 and 23
respectively) in an original video recorded in a park. The second and third
rows show the stylized frames obtained by SFNs trained without temporal loss
and with the P-FDB loss, respectively. The style image is “La Muse”. See
Supplementary Video 4.

####  4.5.2 Video stability: quantitative results

Throughout the paper, if two methods A and B are compared in a 2AFC
experiment, we use A$\leftarrowtail$B to denote the percentage of votes for
choosing A.

The evaluation result of video stability is shown in Table 8(a). First, with
either the OFB loss or the P-FDB loss, the model generated more stable videos
than without any temporal loss, indicating that these two temporal losses
effectively improved video stability. Second, the P-FDB loss won approximately
64.1%-50.0%=14.1% votes against the OFB loss, indicating that the P-FDB loss
improved stability more than the OFB loss. Third, there was a non-negligible
portion of votes for no temporal loss (21.7% for the OFB loss and 24.6% for
the P-FDB loss), indicating that in some cases the difference in the stability
was not significant.

TABLE I: Evaluation results of the SFNs with the P-FDB loss, the OFB loss and
without temporal loss (“None”).

| P-FDB $\leftarrowtail$ OFB | OFB $\leftarrowtail$ None | P-FDB $\leftarrowtail$ None
---|---|---|---
Composition | 70.7% | 80.0% | 79.2%
Feathers | 62.3% | 74.3% | 66.8%
La Muse | 57.8% | 83.5% | 78.3%
Starry Night | 65.8% | 73.5% | 77.5%
Average | 64.1% | 78.3% | 75.4%
(a) Results for video stability.

| P-FDB $\leftarrowtail$ OFB | OFB $\leftarrowtail$ None | P-FDB $\leftarrowtail$ None
---|---|---|---
Composition | 46.0% | 40.0% | 34.2%
Feathers | 41.3% | 39.8% | 20.3%
La Muse | 36.3% | 55.0% | 30.3%
Starry Night | 30.3% | 25.0% | 11.3%
Average | 38.6% | 40.7% | 24.0%
(b) Results for frame stylization quality.

####  4.5.3 Frame stylization quality: qualitative results

We visually inspected the results and found that both the P-FDB loss and the
OFB loss showed stylization quality degradation. As shown in Fig. 5, both the
P-FDB loss and the OFB loss smoothed out the stylization texture to some
extent. This phenomenon is clearly observed in the left example in Fig. 5. As
indicated by red boxes, with these temporal losses, the sharp stripes
presented in the results obtained without any temporal loss were largely
suppressed. The stylization quality degradation is not clear in the right
example in Fig. 5.

In principle it is possible that the coefficient for the temporal loss was too
large. However, we found that the stylized videos became unstable when we
tried to decrease the P-FDB loss coefficient $\lambda_{3}$.

####  4.5.4 Frame stylization quality: quantitative results

We evaluated the stylization quality of the results obtained with different
temporal losses and without any temporal loss by conducting 2AFC experiments
as described in 4.3. The results are presented in Table 8(b). Both the P-FDB
loss and the OFB loss received significantly fewer votes than no temporal
loss, indicating that both temporal losses degraded the stylization quality.
Compared with no temporal loss, the votes received by the P-FDB loss were
fewer than the votes received by the OFB loss, indicating that the P-FDB loss
affected the stylization quality more than the OFB loss. A direct comparison
between the two temporal losses verified this conclusion. The P-FDB loss
received only 38.6% votes while the OFB loss received 61.4% votes.

The stylization quality degradation caused by the P-FDB loss may be due to its
aggressive requirement to stabilize the video, i.e., making the pixel-to-pixel
difference in the stylized video frames similar to that of the original video
frames. According to (5) and (6), when $l=0$,

$L_{\text{temp}}\propto\sum_{t=1}^{T-1}\left\lVert(\tilde{I}_{t+1}-I_{t+1})-(\tilde{I}_{t}-I_{t})\right\rVert_{2}^{2}.$
(10)

The minimum is attained if $\tilde{I}_{t}-I_{t}$ is a constant for all $t$.
The stylization image will be close to the original image in this case.
Although the style loss counteracts this trend, the stylization quality may be
degraded.

The stylization quality degradation caused by the OFB loss may be due to the
complexity of the videos in the DAVIS dataset. According to the official
statistics [36], 60% of videos have complicated deformations, 42% of videos
contain fast-moving objects (per-frame object motion is larger than 20 px),
and 36% of videos contain occluded objects and other difficult cases. As
mentioned in Section 1, these scenarios are challenging for optic flow
estimation and will lead to inaccurate optical flow estimation. It is
difficult to design a better optic flow estimation algorithm to avoid
stylization quality degradation of the OFB loss, because this task is also
quite challenging.

###  4.6 Results with the C-FDB loss

The previous experiments showed that the P-FDB loss led to style quality
degradation. As discussed above, we suspect that the constraint of enforcing a
pixel-level frame difference is too strong. This constraint is relaxed to some
extent if we define the FDB loss in the feature space, i.e., $L_{\text{ft}}$.
The training pipeline is illustrated in Fig. 2(b). We note that if $l\neq 0$
in (5) and (6),

$L_{\text{temp}}\propto\sum_{t=1}^{T-1}\left\lVert(f_{l}(\tilde{I}_{t+1})-f_{l}(I_{t+1}))-(f_{l}(\tilde{I}_{t})-f_{l}(I_{t}))\right\rVert_{2}^{2}.$
(11)

The minimum is attained if $f_{l}(\tilde{I}_{t})-f_{l}(I_{t})$ is a constant
for all $t$. However, this does not imply that $\tilde{I}_{t}-I_{t}$ must be a
constant due to the nonlinear transformations (e.g., max pooling) in the
neural network from $x$ to $f_{l}(x)$. Considering that tiny flickering may be
neglected by $L_{\text{ft}}$, we combined $L_{\text{ft}}$ with $L_{\text{pt}}$
to form the C-FDB loss $L_{\text{ft}}$. In this subsection, we present the
results of the C-FDB loss.

####  4.6.1 Video stability: qualitative results

(c) Stylized using the “Composition VII” image.

(d) Stylized using the “Starry Night” image.

Figure 8: Stylization results of the SFNs trained without temporal loss, with
the OFB loss, the P-FDB loss and the C-FDB loss. See Supplementary Video 5.

We compared the results obtained using the SFNs trained with the C-FDB loss,
the P-FDB loss, the OFB loss and without any temporal loss. Several stylized
videos are shown in Supplementary Video 5. Two successive frames stylized by
SFNs trained with three temporal losses and without temporal loss are shown in
Fig. 8. Without temporal loss, the two frames displayed large difference as
highlighted by green boxes, leading to flickers in the videos. However, the
flickers were largely reduced in the results obtained with temporal losses.

####  4.6.2 Video stability: quantitative results

We conducted 2AFC experiments about video stability with the results listed in
Table III(a). First, the C-FDB loss received 72.9% votes compared with no
temporal loss, indicating that the C-FDB loss successfully improved the video
stability. Second, the C-FDB loss received fewer votes than the P-FDB loss,
indicating that the stabilization ability of the C-FDB loss was lower than
that of the P-FDB loss. Third, the difference of the votes between the C-FDB
loss and the OFB loss was within 2%, indicating that the two losses had
similar capability to stabilize stylized videos.

TABLE II: Evaluation results of the SFNs with the C-FDB loss, the P-FDB loss,
the OFB loss and without temporal loss (“None”)

Method | C-FDB $\leftarrowtail$ P-FDB | C-FDB $\leftarrowtail$ OFB | C-FDB $\leftarrowtail$ None
---|---|---|---
Composition | 37.2% | 50.5% | 77.2%
Feathers | 48.0% | 49.7% | 71.0%
La Muse | 39.5% | 40.5% | 75.5%
Starry Night | 34.3% | 51.5% | 68.0%
Average | 39.8% | 48.1% | 72.9%
(a) Results for video stability.

Method | C-FDB $\leftarrowtail$ P-FDB | C-FDB $\leftarrowtail$ OFB | C-FDB $\leftarrowtail$ None
---|---|---|---
Compositions | 50.2% | 40.7% | 50.0%
Feathers | 59.8% | 55.2% | 36.3%
La Muse | 67.5% | 50.0% | 47.3%
Starry Night | 68.8% | 57.8% | 31.0%
Average | 61.6% | 50.9% | 41.6%
(b) Results for frame stylization quality.

####  4.6.3 Frame stylization quality: qualitative results

We then compared the stylization quality of the frames obtained with different
temporal losses. A visual inspection showed that the C-FDB loss and the OFB
loss led to richer texture variations in content images than the P-FDB loss.
In the examples shown in Fig. 8, the results obtained with the C-FDB loss and
the OFB loss had similar stylization quality to the result obtained without
temporal loss but displayed clearly higher quality than the result obtained
with the P-FDB loss. In fact, in Fig. 8(c), the faces in the 1st, 2nd and 4th
rows have richer colors than the faces in the 3rd row. In Fig. 8(d), the
bodies of the female pedestrian in the 2nd and the 4th rows have more similar
colors than the 3rd row to the bodies in the 1st row.

####  4.6.4 Frame stylization quality: quantitative results

Quantitative evaluations were performed in the same manner as that used to
compare video stability (see Section 4.6.2). The C-FDB loss was compared with
the P-FDB loss, the OFB loss, and no temporal loss. The results are shown in
Table III(b). Compared with no temporal loss, the C-FDB loss received 41.6%
votes. Compared with the P-FDB loss, the C-FDB loss received 61.6% votes.
Compared with the OFB loss, the C-FDB loss received 50.9% votes. The results
indicated that the stylization quality of the C-FDB loss matched that of the
OFB loss but was higher than that of the P-FDB loss.

###  4.7 Results of the RNN

We trained two RNNs with the C-FDB loss and the OFB loss, respectively. As a
baseline model, the SFN trained without any temporal loss was also included in
comparison. Comparison with this baseline model is meaningful because this
model was used as the initial model for training the RNNs (see Section 4.2).
We performed 2AFC experiments with the same setting as described in Section
4.3.

####  4.7.1 Qualitative results

Several example videos stylized by the RNNs are shown in Supplementary Video
6. Fig. 9 shows two successive frames from different videos. For video
stability, as expected, the two frames obtained by the SFN trained without
temporal loss were inconsistent as highlighted by blue boxes. In Fig. 9(a),
the shape and the number of stripes are clearly different in the two frames.
In Fig. 9(b), the black-white texture is sharper in the second frame. The two
frames output by the RNN trained with the OFB loss or the C-FDB loss were more
consistent. In fact, the videos produced by the two RNNs were more stable (see
Supplementary Video 6). In terms of stylization quality, the results of the
two RNNs were hardly distinguishable.

(a) Stylized using the “Feathers” image

(b) Stylized using the “Composition VII” image

Figure 9: Stylization results of the SFN trained without any temporal loss
(top), and the results of the RNNs trained with the OFB loss (middle) and the
C-FDB loss (bottom). See Supplementary Video 6.

####  4.7.2 Quantitative results

TABLE III: Evaluation results of the RNNs trained with the C-FDB loss, the OFB
loss and the results of the SFN trained without any temporal loss (“None”).

Method | C-FDB $\leftarrowtail$ OFB | C-FDB $\leftarrowtail$ None
---|---|---
Composition | 74.7% | 70.0%
Feathers | 59.3% | 67.3%
La Muse | 49.3% | 75.5%
Starry Night | 58.3% | 71.0%
Average | 60.4% | 72.2%
(a) Results for video stability.

Method | C-FDB $\leftarrowtail$ OFB | C-FDB $\leftarrowtail$ None
---|---|---
Composition | 71.2% | 48.2%
Feathers | 59.8% | 36.8%
La Muse | 53.5% | 51.8%
Starry Night | 57.8% | 36.3%
Average | 60.6% | 43.2%
(b) Results for frame stylization quality.

Tables IV(a) and IV(b) show the 2AFC results for video stability and frame
stylization quality, respectively. For stability, the C-FDB loss received
72.2% votes compared with no temporal loss and 60.4% votes compared with the
OFB loss. For stylization quality, the C-FDB received 43.2% votes compared to
the method with no temporal loss and 60.6% votes compared to the method with
the OFB loss. These results suggested that the C-FDB loss was a better choice
than the OFB loss for training the RNN.

Compared with the SFN trained without any temporal loss, the RNN trained with
the C-FDB loss (Table III) and the SFN trained with the C-FDB loss (Table II)
performed equally well in terms of either video stability or frame stylization
quality. However, the results of the comparison between the C-FDB loss and the
OFB loss showed that the C-FDB loss was better on the RNN than on the SFN. In
fact, on the RNN, the C-FDB received 60.4% votes for video stability compared
with the OFB loss (Table IV(a)), but this number was 48.1% on the SFN (Table
III(a)). On the RNN, the C-FDB received 60.6% votes for frame stylization
quality when compared with the OFB loss (Table IV(b)), but this value was
50.9% on the SFN (Table III(b)).

###  4.8 Summary of the experimental results

We summarize the experimental results presented in this section:

  * •

On the SFN, the P-FDB loss produced better stability results than the OFB
loss, but both losses suffered from stylization quality degradation when
compared with no temporal loss.

  * •

On the SFN, the C-FDB loss was better than the P-FDB loss in terms of
stylization quality.

  * •

On the SFN, the C-FDB loss was comparable to the OFB loss in terms of
stability and stylization quality.

  * •

On the RNN, the C-FDB loss was better than the OFB loss in terms of stability
and stylization quality.

We note that both the SFN and the RNN were significantly faster in training
with the C-FDB loss than with the OFB loss, because the C-FDB loss did not
require the time-consuming estimation of optical flows for videos.

##  5 Concluding Remarks

The cutting-edge video stylization models usually adopt OFB methods to
stabilize frames. However, the calculation of optical flow is expensive and
challenging in complex scenarios. Since stability is encoded in the frame
difference in the original video, we propose to stabilize the frames by
requiring the output frame difference to be close to the input frame
difference. The FDB loss is formulated and can be calculated either in the
pixel space, or in the feature space of certain deep neural networks, or in
both spaces. This technique is simple yet effective. We compared the
performances of the proposed loss and the popular OFB temporal loss on two
typical video stylization networks. Extensive human behavior experiments
showed that the proposed loss was comparable to the OFB loss on one network
and better than the OFB loss on the other network in terms of stability and
stylization quality of the videos.

The proposed FDB loss can be applied to most video stylization models that use
the OFB loss. In addition, it may have applications beyond video stylization.
Indeed, in any task that requires temporal coherence between the video frames
while the original stable video is known, e.g., color grading [35] and tonal
stabilization [27], the proposed frame difference-based method or its
extension may be a good substitute for the optic flow-based methods. Using the
OFB loss means that users must understand and correctly set up the optic flow
estimation pipeline, requiring an intense effort in practice. By contrast, the
C-FDB loss can be applied to new models similar to any other loss functions
used to train neural networks. It does not require time-consuming
preprocessing. We believe that the proposed method will be widely applied in
video processing.

## Acknowledgments

The authors would like to thank Yueqiao Li and Jian Wu for helping to set up
the experiments. This work was supported in part by the National Natural
Science Foundation of China under Grant 62061136001, Grant 61836014, Grant
U19B2034 and Grant 61620106010.

## References

  * [1] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2414–2423.
  * [2] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European Conference on Computer Vision_. Springer, 2016, pp. 694–711.
  * [3] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. S. Lempitsky, “Texture networks: Feed-forward synthesis of textures and stylized images.” in _Proceedings of the International Conference on Machine Learning_ , 2016, pp. 1349–1357.
  * [4] V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” in _International Conference on Learning Representations_ , 2017\.
  * [5] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang, “Diversified texture synthesis with feed-forward networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3920–3928.
  * [6] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1501–1510.
  * [7] D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua, “Stylebank: An explicit representation for neural image style transfer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1897–1906.
  * [8] J. L. Barron, D. J. Fleet, and S. S. Beauchemin, “Performance of optical flow techniques,” _International Journal of Computer Vision_ , vol. 12, no. 1, pp. 43–77, 1994.
  * [9] S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski, “A database and evaluation methodology for optical flow,” _International Journal of Computer Vision_ , vol. 92, no. 1, pp. 1–31, 2011.
  * [10] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2758–2766.
  * [11] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2462–2470.
  * [12] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, “Deepflow: Large displacement optical flow with deep matching,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2013, pp. 1385–1392.
  * [13] D. Kotovenko, A. Sanakoyeu, P. Ma, S. Lang, and B. Ommer, “A content transformation block for image style transfer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 032–10 041.
  * [14] X. Li, S. Liu, J. Kautz, and M.-H. Yang, “Learning linear transformations for fast image and video style transfer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3809–3817.
  * [15] M. Ruder, A. Dosovitskiy, and T. Brox, “Artistic style transfer for videos,” in _German Conference on Pattern Recognition_. Springer, 2016, pp. 26–36.
  * [16] H. Huang, H. Wang, W. Luo, L. Ma, W. Jiang, X. Zhu, Z. Li, and W. Liu, “Real-time neural style transfer for videos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 783–791.
  * [17] M. Ruder, A. Dosovitskiy, and T. Brox, “Artistic style transfer for videos and spherical images,” _International Journal of Computer Vision_ , vol. 126, no. 11, pp. 1199–1219, 2018.
  * [18] A. Gupta, J. Johnson, A. Alahi, and L. Fei-Fei, “Characterizing and improving stability in neural style transfer,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4067–4076.
  * [19] O. Frigo, N. Sabater, J. Delon, and P. Hellier, “Split and match: Example-based adaptive patch sampling for unsupervised style transfer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 553–561.
  * [20] ——, “Video style transfer by consistent adaptive patch sampling,” _The Visual Computer_ , vol. 35, no. 3, pp. 429–443, 2019.
  * [21] W. Wang, S. Yang, J. Xu, and J. Liu, “Consistent video style transfer via relaxation and regularization,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 9125–9139, 2020.
  * [22] D. Chen, J. Liao, L. Yuan, N. Yu, and G. Hua, “Coherent online video style transfer,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1105–1114.
  * [23] N. Kong, P. V. Gehler, and M. J. Black, “Intrinsic video,” in _European Conference on Computer Vision_. Springer, 2014, pp. 360–375.
  * [24] G. Ye, E. Garces, Y. Liu, Q. Dai, and D. Gutierrez, “Intrinsic video and applications,” _ACM Transactions on Graphics_ , p. 88, 2014.
  * [25] N. Bonneel, K. Sunkavalli, J. Tompkin, D. Sun, S. Paris, and H. Pfister, “Interactive intrinsic video editing,” _ACM Transactions on Graphics_ , vol. 33, no. 6, p. 197, 2014.
  * [26] P. Litwinowicz, “Processing images and video for an impressionist effect,” in _Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques_ , 1997, pp. 407–414.
  * [27] Z. Farbman and D. Lischinski, “Tonal stabilization of video,” in _ACM Transactions on Graphics_ , vol. 30, no. 4, 2011, p. 89.
  * [28] N. Bonneel, J. Tompkin, K. Sunkavalli, D. Sun, S. Paris, and H. Pfister, “Blind video temporal consistency,” _ACM Transactions on Graphics_ , vol. 34, no. 6, p. 196, 2015.
  * [29] W.-S. Lai, J.-B. Huang, O. Wang, E. Shechtman, E. Yumer, and M.-H. Yang, “Learning blind video temporal consistency,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 170–185.
  * [30] X. Wei, J. Zhu, S. Feng, and H. Su, “Video-to-video translation with global temporal consistency,” in _ACM Conference on Multimedia_. ACM, 2018, pp. 18–25.
  * [31] A. Hosni, C. Rhemann, M. Bleyer, and M. Gelautz, “Temporally consistent disparity and optical flow via efficient spatio-temporal filtering,” in _Pacific-Rim Symposium on Image and Video Technology_ , 2011, pp. 165–177.
  * [32] M. Lang, O. Wang, T. O. Aydin, A. Smolic, and M. H. Gross, “Practical temporal consistency for image-based graphics applications.” _ACM Transactions on Graphics_ , vol. 31, no. 4, pp. 34–1, 2012.
  * [33] S. Paris, “Edge-preserving smoothing and mean-shift segmentation of video streams,” in _European Conference on Computer Vision_. Springer, 2008, pp. 460–473.
  * [34] A. Meka, M. Zollhöfer, C. Richardt, and C. Theobalt, “Live intrinsic video,” _ACM Transactions on Graphics_ , vol. 35, no. 4, p. 109, 2016.
  * [35] N. Bonneel, K. Sunkavalli, S. Paris, and H. Pfister, “Example-based video color grading.” _ACM Transactions on Graphics_ , vol. 32, no. 4, pp. 39–1, 2013.
  * [36] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, “A benchmark dataset and evaluation methodology for video object segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [37] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in _European conference on computer vision_. Springer, 2012, pp. 611–625.
  * [38] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014.
  * [39] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [41] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 586–595.


(SiC)n ionization]The vertical and adiabatic ionization energies of silicon carbide clusters, (SiC)n, with n=1-12

Silicon carbide (SiC) is one of the major cosmic dust components in carbon-rich environments.
However, the formation of SiC dust is not well understood. In particular, the initial stages of the SiC condensation (i.e. the SiC nucleation) remain unclear, as the basic building blocks (i.e. molecular clusters) exhibit atomic segregation at the (sub-)nanoscale.
We report vertical and adiabatic ionization energies of small silicon carbide clusters, (SiC)n, n=2$-$12, ranging from 6.6$-$10.0 eV, which are lower than for the SiC molecule ($\sim$10.6 eV).
The most favorable structures of the singly ionized (SiC)n+, n=5$-$12, cations resemble their neutral counterparts. However, for sizes n=2$-$4, these structural analogues are metastable and different cation geometries are favored. Moreover, we find that the (SiC)5+ cation is likely to be a transition state.

Therefore, we place constraints on the stability limit for small, neutral (SiC)n clusters to persist ionization through (inter)-stellar radiation fields or high temperatures.

§ KEYWORDS:
silicon carbide, SiC, ionization, clusters, cations, nucleation, dust formation, circumstellar

§ INTRODUCTION

The chemistry occurring in astrophysical environments is primarily controlled by the carbon-over-oxygen (C/O) ratio. This is a consequence of the high carbon monoxide (CO) bond energy of 11.2 eV making CO the most stable diatomic molecule known [Luo, 2007].
This classical dichotomy is challenged since non-equilibrium processes such as photo-chemistry and
pulsation-induced shocks can break the strong CO bond and lead to the formation of unexpected molecules [Agúndez et al., 2010, Gobrecht et al., 2016]. Of particular astronomical interest are refractory molecules and molecular clusters that represent the precursors of circumstellar dust.
One of the main dust species in carbon-dominated regimes is silicon carbide (SiC).
A broad spectral feature around 11.3 micron is commonly observed in Carbon-rich evolved starts and attributed to the presence of SiC dust grains [Friedemann, 1968, Hackwell, 1972, Treffers and Cohen, 1974].
SiC stardust was extracted from pristine meteorites [Bernatowicz et al., 1987, Amari et al., 1994, Hoppe et al., 1996, Zinner et al., 2007, Liu et al., 2014]. Recent studies have shown that the vast majority of presolar SiC grains found in pristine meteoritic stardust originate from low-mass Asymptotic Giant Branch (AGB) stars [Cristallo et al., 2020]. But also molecular gas-phase species like SiC, Si2C, SiC2 were detected in the circumstellar envelopes of carbon-rich evolved stars [Cernicharo et al., 1989, McCarthy et al., 2015, Thaddeus et al., 1984, Massalkhi, S. et al., 2018]. The evidence for both, gas-phase silicon-carbon molecules and solid SiC dust suggests that their intermediaries, i.e. SiC molecular clusters, also exist in carbon-rich astronomical environments and partake in the nucleation and SiC dust formation process.

Therefore, SiC molecular clusters are the objects of our interest.
This study represents a continuation of a prior work [Gobrecht et al., 2017] and adresses the (single) ionization energies of the previously investigated neutral (SiC)n, n=1-12, clusters.
This paper is organized as follows.
In Section 2 we present the methods used to derive the vertical and adiabatic ionization energies. Section 3 shows the results for these energies as well as the adiabatically optimized cation geometries and in Section 4 our summary and conclusion is given.

§ METHODS

In this study, we focus on small, singly ionized silicon carbide cation clusters with a silicon-to-carbon stoichiometry of 1:1, (SiC)n+.

The smallest lowest-energy neutral (SiC)n, n=1$-$11, clusters are characterized by atomic segregation of carbon and silicon [Hou and Song, 2008, Duan et al., 2013, Byrd et al., 2016]. The carbon atoms tend to form chains and benzene-like rings, whereas the silicon atoms
develop triangular and tetrahedral substructures bridging to the carbon segregation. The presence of aromatic carbon rings in the neutral (SiC)n Global Minima (GM) candidates indicates the occurence of strongly bound, delocalised valence electrons .

We investigate the vertical and adiabatic single-electron ionization energies of (SiC)n, n=1$-$12, clusters by means of Density Functional Theory (DFT). The energies and geometries of the lowest-energy neutral isomers (i.e. GM candidates) were already presented in a previous study [Gobrecht et al., 2017].
These calculations were performed on the M11/cc-pVTZ level of theory [Peverati and Truhlar, 2012], which was shown to result in accurate energies with affordable computational cost and, in contrast to the B3LYP functional, is not outperforming [Byrd et al., 2016].
For consistency and comparability, the (SiC)n+ cluster calculations were also performed on the same level of theory (M11/cc-pVTZ). For the DFT calculations we use the computational software Gaussian 09 [Frisch et al., 2009].
For all optimizations a vibrational frequency analysis is performed. First, it allows us to discriminate between Transition States (TSs) exhibiting one imaginary frequency and true minima with only real vibration modes. In the case of a TS, marking a saddle point on the potential energy surface, the geometry is distorted
in direction of the negative curvature of the saddle point and re-optimized to obtain a real minimum.
Second, partition functions can be constructed to calculate thermodynamic potentials within the Rigid Rotor Harmonic Oscillator (RRHO) appromixation.
Third, the resulting vibrational spectra can be compared with observational data.

The current work represents an extension of a previous (SiC)n cluster study [Gobrecht et al., 2017], which did not comprise the ionization of these clusters and its related cations.

Owing to the enormous computational task, we did not perform global optimization searches of the (SiC)n+ cations themselves.
Instead, we perform ionization calculations for the four lowest-lying neutral isomers for each size n.
As mentioned in [Gobrecht et al., 2017], the GM candidate isomer may not be the most favorable structure in circumstellar conditions and hence it is required to study a range of the energetically lowest-lying structures and their respective cations for each cluster size.
Therefore, we assume that the neutral (SiC)n clusters are initially present in the form of their four lowest-energy isomers including their GM candidate structure, and that they are subsequently ionized.
We are aware that our set of isomers is not complete and hence, there is the possibility that (SiC)n cation isomers with a lower energy exist.
In order to reduce this possibility and to avoid missing a particularly stable SiC cluster cation, additional test calculations for different geometries (e.g. cages and tubes) are performed.

It is known that neutral molecules and related clusters can alter their ground-state geometry significantly upon the loss of an electron. They include, for example, clusters of water [Belau et al., 2007] and Argon [Briant et al., 2012], but also organic compounds like methyl ketene [Derbali et al., 2020], acetic acid [Guan et al., 2012] or $\delta$-Valerolactam [Mahjoub et al., 2011].
Moreover, the spin-degenerate cluster cations can be subject to Jahn-Teller effects implying a lowering of their spatial symmetry and a structural rearrangement. Therefore, it is instructive to not only study the vertical ionization energies, corresponding to single point energy evaluations, but also adiabatic ionization energies taking into account structural rearrangements.
Here, all cations are presumed to be in a doublet state (i.e. with a spin multiplicity of 2), since the neutral (SiC)n, n$\ge$2, clusters are singlet states and the SiC molecule (n=1) is in a triplet state.

§ RESULTS

In the following, we present the vertical and adiabatic ionization energies of (SiC)n, n=1$-$12, clusters and the related structures of the adiabtically optimized, singly ionized (SiC)n+, n=1$-$12, cations.

§.§ Vertical ionization

The vertical ionization energies, Envert, are listed in Table <ref>.
Envert of the lowest-energy cations (denoted with GM) show an overall monotonically decreasing trend with cluster size n, starting from 10.64 eV (n=1) to 6.69 eV (n=12), with opposing tendencies at n=5 and n=9.
For the smallest sizes (n=1$-$2) Envert decrease more strongly than for larger sizes. For the latter size regime the decrease in Envert flattens, except for n=9, where E9vert=7.34 is slightly larger than for n=8 and n=10. Considering only structural analogues of the neutral GM clusters (denoted as LM), we find a similar decreasing trend with cluster size n, except for n=9.

§.§ Adiabatic ionization

In this subsection, we discuss the adiabatic ionization
energies Enadia and compare their respective geometries with those of the neutral clusters.
In Figure <ref> the lowest-energy cation structures are shown. Cation isomers that are derived from neutral GM candidates (i.e. structural twins or analogues), but which are metastable with respect to the lowest-energy cations, are depicted in Figure <ref>.
For comparison we also display the neutral GM candidates in Figure <ref> and local minima neutral isomers (see Figure <ref>) corresponding to the structural analogues of the lowest-energy cations.

The triplet ground state of the neutral SiC molecule has a bond length of 1.707 Å (see Figure <ref>). The adiabatically optimized SiC+ cation is assumed to be a doublet state and has a slightly larger bond length of 1.713 Å. The energy difference of vertical and adiabatic ionization is negligible (0.002 eV).
In comparison to the experimentally derived ionization energy of 9.2 eV [Drowart et al., 1958] and to the value of 8.7$\pm$0.2 from a previous theoretical investigation [Boldyrev et al., 1994], we obtain a higher value (10.64 eV) for the SiC ionization potential.

For the SiC dimer, (SiC)2, a linear Si-C-C-Si geometry represents the lowest-energy cation structure (see Figure <ref>), which is different from the diamond shaped neutral GM candidate shown in Figure <ref>.
In comparison with the linear geometry of the neutral dimer (see Figure <ref>), the two Si-C bonds in the cation are larger by 0.055 Å and the C-C bond increases by 0.031 Å.
The adiabatic ionization energy is 7.81 eV with respect to the neutral GM candidate and 7.52 eV with respect to the neutral Si-C-C-Si chain.
The rhombic (diamond-shaped) (SiC)2 cation shown in Figure <ref>, resembling the neutral GM candiate, lies 9.87 eV above the latter. In this cation, the C-C bond distance is increased from 1.421 Å to 1.537 Å and the four Si-C bonds are decreased from 1.827 Å to 1.809 Å, when adiabatically ionized. This indicates that a double bonded C=C in the neutral (SiC)n becomes single bonded in the related cation.

Other isomers show ionization energies in the range of 8.48$-$14.05 eV.
For n=3, the lowest-energy cation in our set corresponds to a structurally different isomer than the neutral GM (see Figure <ref> and <ref>). It has an energy of 7.94 eV above the neutral GM and 7.04 eV above its neutral structural twin. The largest geometric change upon the loss of an electron consists in the breaking of a bond (marked with dashed line and labelled with a distance of 2.953 Å in Figure <ref>), which is present in the neutral isomer (2.047 Å).
The other bond lengths change by 0.03$-$0.11 Å.
The cationic twin of the neutral GM displayed in Figure <ref> has an energy 1.01 eV above the lowest-energy cation and is a TS, whose distorted, asymetric geometry results in a real minimum with a relative energy of 0.88 eV.
These cations show increased Si-Si distances as compared with the neutral GM.
Further investigated trimer isomers have ionization energies in the range of 8.20$-$10.26 eV.

The adiabatic ionization of different (SiC)4 isomers shows that the lowest-energy cation is a structural analogue of the second most favourable neutral structure (see Figure <ref> and <ref>). The GM cation has an adiabatic ionization energy of 7.36 eV with respect to the neutral GM and 7.07 eV with repect to the neutral structure analogue.
It consists of a C4 chain in a trans configuration with two Si atoms arranged at each of the two tails of the chain. Upon the loss of an electron, most significantly, the two Si-Si bonds increase by 0.13 Å among minor changes in the other bond lengths.
As for n=3, the optimized (SiC)4+ structural twin of the neutral GM shows an imaginary vibrational frequency and is a TS with an energy of 8.63 eV above the neutral GM.
To obtain a real minimum, a further optimization is performed leading to a symmetry breaking of the C2-Si-C2-Si ring structure by increasing two Si-C bond lengths and by tilting the quasi planar ring structure. The local minimum lies about 0.40 eV below the TS. Other cation isomers show ionization energies of 7.41$-$7.84 eV

The most favourable (SiC)5+ cation resembles its neutral analogue and has an ionization energy of 7.61 eV. However, in contrast to its neutral counterpart the two out-of-plane Si atoms increase their distance to each other by 0.25 Å.
This structure is a TS and a subsequent optimization to a real minimum leads (as for n=3 and 4) to a breaking of the C2v symmetry. In the real minimum cation, the two out-of-plane Si atoms have distance of 2.63 Å to each other (like in the neutral cluster), but are shifted into the display plane, which reduces the energy by 0.14 eV (as compared to the TS).
The ionization energies of other investigated isomers range from 7.90 eV to 8.32 eV

For n=6, the lowest-energy cation is a structural analogue of the neutral GM candidate with an ionization energy of 7.46 eV.
Its overall geometry hardly changes by the adiabatic ionization. The C-C and Si-C bond distances are largely preserved. The biggest change occurs at two Si atoms increasing their distance by 0.30 Å. This likely to be the place, where the loss of the electron occured during the adiabatic ionization, which accounts for 0.44 eV. We find ionization energies of 7.50$-$8.45 eV for other hexamer cation isomers.

The lowest-energy (SiC)7+ cation resembles the neutral (SiC)7 GM cluster and lies 7.25 eV above it. The largest changes in the bond distances occur at Si-Si bonds that lie out-of-plane. The difference in the vertical and adiabatic ionization energy (0.31 eV) is smaller than for n=6 and n=8. Further (SiC)7+ isomers exhibit ionization energies of 7.60$-$8.26 eV.

For n=8, an ionization energy of 7.25 eV is found for the lowest-energy cation, which is a look-alike of the neutral GM candidate.
The largest atomic mobility during the adiabatic ionization is accomplished by the out-of-plane Si atoms.
These two Si atoms have moved closer to the C atom and are now 4-fold coordinated, in comparison to the neutral (SiC)8 cluster, where these Si atoms are 3-fold coordinated. The adiabatic structure change results in a lower energy of 0.37 eV. For other isomers we find ionization energies of 7.62$-$7.73 eV.

For n=9, the neutral GM and lowest-energy cation exhibit a similar shape and differ by 6.96 eV in energy.
Its adiabatic ionization affects predominantly the silicon-segregated part of the isomer. The two Si atoms out-of-plane increase their distance to each other, whereas two Si atoms in the plane move closer to each other accounting for 0.38 eV. Other local minima have ionization energies of 7.11$-$7.64 eV.

For n=10, both the neutral (SiC)10 clusters and its cation (SiC)10+, do not exhibit planar carbon ring structures, but are strained. This is the smallest cluster / cation size, where the C atoms do not lie in a planar configuration.
The lowest-energy (SiC)9+ cation geometry corresponds to the one of the neutral GM candidate and has a relative energy of 6.95 eV.
The largest impact on the geometry is found to be located at a single Si atom. In the cation this Si atom is connected to the silicon subcluster, whereas in the neutral cluster it binds only to C atoms. This Si migration causes an energy difference of 0.25 eV. Other cation isomers show ionization energies of 7.30$-$7.84 eV.

For the neutral (SiC)11 GM candidate, the adiabatic optimization of the corresponding cation failed. However, we found a very similar neutral (SiC)11 cluster with a relative energy just 0.10 eV above the GM candidate, for which an adiabatic ionization optimization could be achieved.
With an ionization energy of 6.64 eV it also corresponds to the lowest-energy cation configuration for n=11.
By inspecting the bond lengths of the neutral and cationic (SiC)11 structure,
we find that none of them changes significantly (max of 0.07 Å / bond) during the adiabatic ionization, which is reflected in the comparatively low energy difference of 0.18 eV. Higher-lying cation isomers have energies 6.92$-$7.35 eV above the neutral GM.

The GM candidate of neutral (SiC)12 is found to be a bucky-like, spherical structure with strictly alternating Si-C bonds. However, our calculations for vertical and adiabatic ionization energies did not converge.
The ionization of fullerene-shaped clusters is a dynamic and complex problem and requires high-level calculations and analyses [Hrodmarsson et al., 2020]. For example, Buckminster fullerenes cations (C60+) undergo dynamic and pseudo Jahn-Teller effects lowering the icosahedral symmetry upon (photo-)ionization. Owing to these challenges, that are beyond the scope of this paper, we omit an in-depth investigation of the ionization of the “bucky” (SiC)12 GM candiate isomer.
Therefore, we calculate the ionization energies of an atom-segregated symmetric “butterfly”-shaped isomer, which is the second lowest-energy (SiC)12 isomer according to our searches and has a relative energy of 0.54 eV. We find an ionization energy of 7.77 eV and that its geometry is hardly affected by the ionization process (max 0.02 Å / bond ), which also explains the low energy difference (0.10 eV) between vertical and adiabatic ionization.
We investigated two other isomers that result in ionization energies of 7.75 eV and 7.77 eV above the neutral GM.

§ SUMMARY AND CONCLUSIONS

The vertical and adiabatic single ionization energies of (SiC)n, n=1$-$12, span over a range of 6.59$-$10.64 eV and have an overall decreasing trend with cluster size n. In an astrophysical context, this energy range sets constraints for neutral (SiC)n, n=1$-$12, clusters to exist and therefore confines the conditions (temperature, radiation field) under which SiC nucleation can occur.
The adiabatically optimized cation geometries indicate that the ionization predominantly takes place in the silicon-segregated part of t/he clusters.
For n=2,3,4 the lowest-energy cation does not correspond to the structural analogue of the neutral GM candidate structures, but to a different geometry. For these sizes, the structural analogue cations of the neutral GM show potential energies 0.83-1.66 eV and are hence metastable with respect to lowest-energy cation.
Moreover, the optimization of the neutral GM structural analogues for sizes n=3,4,5 leads to transition states (TSs) that are subsequently re-optimized and real minima are found.

Applying the kinetic gas theory to the presented range of ionization energies (6.59$-$10.64 eV) results in equivalent temperatures of $\sim$76500$-$123500 K. These temperatures are significantly higher than the characteristic temperatures in envelopes and winds of evolved stars, even when including non-equilibrium effects like shocks or other non-thermal heating processes.
However, considering photo-ionization instead of ionization by a high temperature gas,the situation is different. For planetary nebulae with effective temperatures of $\sim$ 30000 K, the radiation field peaks for photon energies in the range 6 to 10 eV [Cami et al., 2010]. This photon energy range is almost superimposable with the range of (SiC)n ionization energies derived in this study.
Therefore, we conclude that the ionization of (SiC)n clusters in low-mass AGB stars is viable, in particular towards the end of their lives,
and that photo-ionization is most likely its responsible process.

Vertical and adiabatic ionization energies, Envert and Enadia(in eV), and their difference $\Delta$E=Envert-Enadia, of (SiC)n clusters versus cluster size n.LM
corresponds to a Local Minimum cation geometry, GM to a Gloabl Minimum cation geometry, and TS to a Transition State, respectively.

n     Envert     Enadia     $\Delta$E

1          10.64     10.64     0.00

2 LM     10.01      9.87     0.14

2 GM      8.27      8.21     0.07

3 TS      9.34      8.95     0.39

3 LM      9.34      8.77     0.57

3 GM      8.17      7.94     0.22

4 TS      8.78      8.63     0.15

4 LM      8.78      8.23     0.55

4 GM      7.62      7.36     0.27

5 TS      7.94      7.75     0.18

5 GM      7.94      7.61     0.33

6           7.90      7.46     0.44

7           7.56      7.25     0.31

8           7.22      6.85     0.37

9           7.34      6.96     0.38

10          7.20      6.95     0.25

11          6.82      6.64     0.18

12          6.69      6.59     0.10

§ CONFLICT OF INTEREST STATEMENT

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

§ FUNDING

We aknowledge support from the ERC Consolidator Grant "AEROSOL - Astrochemistry of Old Stars: direct probing of unique chemical laboratories 2016-2020 (PI Leen Decin)", Project number 646758.

§ ACKNOWLEDGMENTS

We acknowledge the CINECA award under the ISCRA initiative, for the availability of high performance computing resources and support.

[Agúndez et al., 2010]
Agúndez, M., Cernicharo, J., and Guélin, M. (2010).
Photochemistry in the inner layers of clumpy circumstellar envelops:
formation of water in C-rich objects and of C-bearing molecules in O-rich
The Astrophysical Journal 724, L133–L136.

[Amari et al., 1994]
Amari, S., Lewis, R. S., and Anders, E. (1994).
Interstellar grains in meteorites. I - Isolation of SiC, graphite,
and diamond; size distributions of SiC and graphite. II - SiC and its noble
Geochimica et Cosmochimica Acta 58, 459.

[Belau et al., 2007]
Belau, L., Wilson, K., Leone, S., and Ahmed, M. (2007).
Vacuum ultraviolet (vuv) photoionization of small water clusters.
The journal of physical chemistry. A 111, 10075–83.

[Bernatowicz et al., 1987]
Bernatowicz, T., Fraundorf, G., Ming, T., Anders, E., Wopenka, B.,
Zinner, E., et al. (1987).
Evidence for interstellar SiC in the Murray carbonaceous meteorite.
Nature 330, 728–730.

[Boldyrev et al., 1994]
Boldyrev, A. I., Simons, J., Zakrzewski, V. G., and von Niessen, W. (1994).
Vertical and adiabatic ionization energies and electron affinities
of new silicon-carbon (SinC) and silicon-oxygen
(SinO) (n = 1-3) molecules.
The Journal of Physical Chemistry 98, 1427–1435.

[Briant et al., 2012]
Briant, M., Poisson, L., Hochlaf, M., de Pujo, P., Gaveau, M.-A., and Soep, B.
${\mathrm{Ar}}_{2}$ Photoelectron Spectroscopy Mediated by
Autoionizing States.
Phys. Rev. Lett. 109, 193401.

[Byrd et al., 2016]
Byrd, J. N., Lutz, J. J., Jin, Y., Ranasinghe, D. S., Montgomery, J. A.,
Perera, A., et al. (2016).
Predictive coupled-cluster isomer orderings for some
SimCn (m,n $<$ 12) clusters: A pragmatic
comparison between DFT and complete basis limit coupled-cluster benchmarks.
The Journal of Chemical Physics 145, 024312.

[Cami et al., 2010]
Cami, J., Bernard-Salas, J., Peeters, E., and Malek, S. E. (2010).
Detection of C60 and C70 in a Young
Planetary Nebula.
Science 329, 1180–1182.

[Cernicharo et al., 1989]
Cernicharo, J., Gottlieb, C. A., Guelin, M., Thaddeus, P., and
Vrtilek, J. M. (1989).
Astonomical and laboratory detection of the SiC radical.
ApJ Letters 341, L25–L28.

[Cristallo et al., 2020]
Cristallo, S., Nanni, A., Cescutti, G., Minchev, I., Liu, N.,
Vescovi, D., et al. (2020).
Mass and metallicity distribution of parent AGB stars of presolar
Astronomy & Astrophysics 644, A8.
2020A   A...644A...8C

[Derbali et al., 2020]
Derbali, I., Hrodmarsson, H. R., Schwell, M., Bénilan, Y., Poisson, L.,
Hochlaf, M., et al. (2020).
Unimolecular decomposition of methyl ketene and its dimer in the gas
phase: theory and experiment.
Phys. Chem. Chem. Phys. 22, 20394–20408.

[Drowart et al., 1958]
Drowart, J., De Maria, G., and Inghram, M. G. (1958).
Thermodynamic study of sic utilizing a mass spectrometer.
The Journal of Chemical Physics 29, 1015–1021.

[Duan et al., 2013]
Duan, X. F., Burggraf, L. W., and Huang, L. (2013).
Searching for Stable SinCn Clusters:
Combination of Stochastic Potential Surface Search and Pseudopotential
Plane-Wave Car-Parinello Simulated Annealing Simulations.
Molecules 18, 8591

[Friedemann, 1968]
Friedemann, C. (1968).
Siliziumkarbid als möglicher Bestandteil des interstellaren
Astronomische Nachrichten 291, 177–186.

[Frisch et al., 2009]
[Dataset] Frisch, M. J., Trucks, G. W., Schlegel, H. B., Scuseria, G. E., Robb,
M. A., Cheeseman, J. R., et al. (2009).
Gaussian 09 Revision A.2

[Gobrecht et al., 2016]
Gobrecht, D., Cherchneff, I., Sarangi, A., Plane, J. M. C., and
Bromley, S. T. (2016).
Dust formation in the oxygen-rich AGB star IK Tauri.
Astronomy & Astrophyiscs 585, A6.
2016A   A...585A...6G

[Gobrecht et al., 2017]
Gobrecht, D., Cristallo, S., Piersanti, L., and Bromley, S. T. (2017).
Nucleation of Small Silicon Carbide Dust Clusters in AGB Stars.
The Astrophysical Journal 840, 117.

[Guan et al., 2012]
Guan, J., Hu, Y., Zou, H., Cao, L., Liu, F., Shan, X., et al. (2012).
Competitive fragmentation pathways of acetic acid dimer explored by
synchrotron VUV photoionization mass spectrometry and electronic structure
The Journal of Chemical Physics 137, 124308.

[Hackwell, 1972]
Hackwell, J. A. (1972).
Long wavelength spectrometry and photometry of M, S and C-stars.
Astronomy & Astrophysics 21, 239
1972A   A....21..239H

[Hoppe et al., 1996]
Hoppe, P., Strebel, R., Eberhardt, P., Amari, S., and Lewis, R. S. (1996).
Small SiC grains and a nitride grain of circumstellar origin from
the Murchison meteorite: Implications for stellar evolution and
Geochimica et Cosmochimica Acta 60, 883 – 907.

[Hou and Song, 2008]
Hou, J. and Song, B. (2008).
Density-functional study of structural and electronic properties of
SinCn(n=1–10) clusters.
The Journal of Chemical Physics 128, 154304.

[Hrodmarsson et al., 2020]
Hrodmarsson, H. R., Garcia, G. A., Linnartz, H., and Nahon, L. (2020).
VUV photoionization dynamics of the C60 buckminsterfullerene:
2D-matrix photoelectron spectroscopy in an astrophysical context.
Phys. Chem. Chem. Phys. 22, 13880–13892.

[Liu et al., 2014]
Liu, N., Savina, M. R., Davis, A. M., Gallino, R., Straniero, O., Gyngard, F.,
et al. (2014).
Barium isotopic composition of mainstream silicon carbides from
murchison: Constraints for s-process nucleosynthesis in asymptotic giant
branch stars.
The Astrophysical Journal 786, 66

[Luo, 2007]
Luo, Y. (2007).
Comprehensive Handbook of Chemical Bond Energies (CRC Press)

[Mahjoub et al., 2011]
Mahjoub, A., Hochlaf, M., Poisson, L., Nieuwjaer, N., Lecomte, F., Schermann,
J.-P., et al. (2011).
Slow photoelectron spectroscopy of d-valerolactam and its dimer.
ChemPhysChem 12, 1822–1832.

[Massalkhi, S. et al., 2018]
Massalkhi, S., Agúndez, M., Cernicharo, J., Velilla Prieto, L.,
Goicoechea, J. R., Quintana-Lacaci, G., et al. (2018).
Abundance of SiC2 in carbon star envelopes -
Evidence that SiC2 is a gas-phase precursor of SiC dust.
Astronomy & Astrophysics 611, A29.

[McCarthy et al., 2015]
McCarthy, M. C., Baraban, J. H., Changala, P. B., Stanton, J. F.,
Martin-Drumel, M.-A., Thorwirth, S., et al. (2015).
Discovery of a missing link: Detection and structure of the elusive
disilicon carbide cluster.
The Journal of Physical Chemistry Letters 6, 2107–2111.
PMID: 26266510

[Peverati and Truhlar, 2012]
Peverati, R. and Truhlar, D. G. (2012).
M11-l: A local density functional that provides improved accuracy for
electronic structure calculations in chemistry and physics.
The Journal of Physical Chemistry Letters 3, 117–124.

[Thaddeus et al., 1984]
Thaddeus, P., Cummins, S. E., and Linke, R. A. (1984).
Identification of the SiCC radical toward IRC +10216 - The first
molecular ring in an astronomical source.
ApJ Letters 283, L45–L48.

[Treffers and Cohen, 1974]
Treffers, R. and Cohen, M. (1974).
High-resolution spectra of cool stars in the 10- and 20-micron
The Astrophysical Journal 188, 545–552.

[Zinner et al., 2007]
Zinner, E., Amari, S., Guinness, R., Jennings, C., Mertz, A.,
Nguyen, A., et al. (2007).
NanoSIMS isotopic analysis of small presolar grains: Search for
Si$_{3}$N$_{4}$ grains from AGB stars and Al and Ti isotopic compositions of
rare presolar SiC grains.
Geochimica et Cosmochimica Acta 71, 4786–4813.

Electronic structures of the lowest-energy (SiC)n+, n=1$-$12, cations. Silicon atoms are displayed in purple, carbon atoms in green. Bond lengths and distances are given in Å. The cluster size n is given at the right bottom of the corresponding structure.

2 LM 1.66 eV

3 TS 1.01 eV

3 LM 0.82 eV

4 TS 1.27 eV

4 LM 0.88 eV

Electronic structures of metastable (SiC)n+, n=2$-$4, cations corresponding to the structural analogues of the neutral Global Minima (GM) clusters. Silicon atoms are displayed in purple, carbon atoms in green. Bond lengths and distances are given in Å. The cluster size n, the type (LM: Local Minimum, TS: Transition State) and the relative energy is given at the right bottom of the corresponding structure.

Electronic structures of neutral (SiC)n, n=1$-$12, global minima (GM) candidates. Silicon atoms are displayed in purple, carbon atoms in green. Bond lengths (in Å) are indiacted by numbers. The cluster size n is given at the right bottom of the corresponding GM structure.

2 LM 0.69 eV

3 LM 0.88 eV

4 LM 0.26 eV

Electronic structures of neutral (SiC)n, n=2$-$4, local minima candidates (denoted as LM) corresponding to the structural analogue of the most favourable (SiC)n+, n=2$-$4, cations.  The cluster size n at the relative energy is given at the right bottom of the corresponding structure.