# To update or not to update?
Neurons at equilibrium in deep models

Andrea Bragagnolo
University of Turin, Italy
Synesthesia s.r.l., Turin, Italy
<EMAIL_ADDRESS>&Enzo Tartaglione
LTCI, Télécom Paris,
Institut Polytechnique de Paris, France
<EMAIL_ADDRESS>&Marco Grangetto
University of Turin, Italy
<EMAIL_ADDRESS>
###### Abstract

Recent advances in deep learning optimization showed that, with some
a-posteriori information on fully-trained models, it is possible to match the
same performance by simply training a subset of their parameters. Such a
discovery has a broad impact from theory to applications, driving the research
towards methods to identify the minimum subset of parameters to train without
look-ahead information exploitation. However, the methods proposed do not
match the state-of-the-art performance and rely on unstructured sparsely
connected models.
In this work we shift our focus from the single parameters to the behavior of
the whole neuron, exploiting the concept of neuronal equilibrium (NEq). When a
neuron is in a configuration at equilibrium (meaning that it has learned a
specific input-output relationship), we can halt its update; on the contrary,
when a neuron is at non-equilibrium, we let its state evolve towards an
equilibrium state, updating its parameters. The proposed approach has been
tested on different state-of-the-art learning strategies and tasks, validating
NEq and observing that the neuronal equilibrium depends on the specific
learning setup.

##  1 Introduction

In recent years, deep learning has become a staple solution to different
tasks, such as computer vision, bio-informatics, speech recognition, and many
more. Unfortunately, modern neural network architectures are becoming more and
more “expensive”: performing simple inferences requires a lot of computational
resources, and training the models even more. Such cost poses several
challenges for the research community: the training of a network model is
associated with large carbon footprints and the commercialization of AI
research (especially for edge devices) is hindered by the resource
requirements of the models Strubell et al. (2019).
For several years now, many works in literature have shown that is possible to
shrink both the size and resource requirements, mainly via quantization Yang
et al. (2020); Jin et al. (2022) and pruning Wang et al. (2020); Tang et al.
(2020); Tanaka et al. (2020). In particular, pruning techniques can
drastically reduce the number of operations needed to perform inference,
without affecting the overall performance of the model. Pruning targets the
reduction of parameters in the model, which has advantages after training;
unfortunately, it is unable to reduce the cost of the training cycle, and on
the contrary, it requires, in general, iterations of few-shot pruning,
followed by fine-tuning Lee et al. (2019); Tartaglione et al. (2018).
A recent work shows the existence of sub-graphs in the whole deep learning
model which, when trained in isolation, match the performance of the whole
model Frankle and Carbin (2019). This opens the road towards a whole field of
research, where many approaches are proposed to find these parameters
a-priori: this shows a potential environmental impact, considering that a
smaller model is trained, leading to a lower energy consumption Strubell et
al. (2019). Indeed, when training a neural network, the back-propagation
procedure and the weights update lead to the larger part of FLOPs (compared to
forward propagation) Plaut et al. (1986); Baydin et al. (2018). Researchers
started to experiment pruning early in the training or even before the
training starts in the hope of training a restricted number of parameters.
Unfortunately, training a sparse network with standard optimizers leads to
subpar results Evci et al. (2019) or the final result does not differ much
from magnitude pruning at the end of the training Frankle et al. (2021). The
causes for such behaviors are still a matter of debate among the community
2019; 2021.
In this work, we shift the focus from the single parameter to the whole
neuron, and we propose NEq, an approach to evaluate whether a given neuron is
at _equilibrium_ for the learning dynamics. If the neuron is in such a state,
its parameters have already reached a target configuration and do not require
a further update. Unlike many other recent approaches, NEq disables _entire
neurons_ (hence, in a structured way), does not require prior knowledge of the
specific task (for example by first training a model to convergence), and
automatically self-adapts to the specific learning policy deployed. Unlike
pruning techniques, NEq does not remove the neurons’ contribution to the
output; instead, it only prevents unnecessary updates to their weights: as a
result, we reduce the number of operations performed by the back-propagation
algorithm and the optimizer.
The rest of the paper is organized as follows. In Sec. 2 we discuss the
related literature; Sec. 3 presents the concept of neuronal equilibrium and
how to evaluate it; Sec. 4 presents the experimental validation inclusive of
an ablation study and Sec. 5 draws the conclusions.

##  2 Related works

It is broadly acknowledged that the typically-deployed deep learning models on
the state-of-the-art scenarios are over-parametrized Mhaskar and Poggio
(2016); Brutzkus et al. (2018). This ignites two lines of research: reducing
the size of these models (with _pruning_ algorithms) or saving computational
resources at training time. While the first has been broadly explored, the
latter suffered a stalemate until a recent work suggested its feasibility
Frankle and Carbin (2019).
Pruning strategies. Attempts to reduce the number of parameters from learned
models date back to 1989 when Mozer and Smolensky proposed _skeletonization_ ,
a technique to identify less relevant neurons in a trained model and to remove
them 1989. This was accomplished thanks to the evaluation of an error function
penalty from a pre-trained model. In the same years, LeCun et al. also
proposed a work where the information from the second-order derivative of the
error function is leveraged to rank the parameters of the trained model on a
saliency-like basis LeCun et al. (1990). In the last decade, thanks to the
broad availability of computational resources, pruning approaches gained-back
popularity, with approaches including the exploitation of dropout Molchanov et
al. (2017); Gomez et al. (2019), sensitivity-based approaches Lee et al.
(2018); Tartaglione et al. (2021, 2022), relaxation of $\ell_{0}$
regularization Louizos et al. (2017); Sanh et al. (2020) and optimization of
auxiliary parameters Xiao et al. (2019). Despite leading to very compact deep
models, the demanded training complexity is frequently very high, as many of
the most well-known approaches rely on iterative fine-tuning (or few-shot
pruning) approaches. Hence, the training complexity for these techniques is
larger than training a vanilla model (which in many cases is used as
initialization). Much computational complexity could be saved if structured
pruning is applied before training the model itself (namely, entire
neurons/channels are pruned). Some works on pruning have suggested the
possibility of re-allocating previously pruned parameters Tresp et al. (1996)
or filters He et al. (2018) to learn new functions and minimize the loss,
towards higher generalization capability Blalock et al. (2020).
The lottery ticket hypothesis. In a recent paper, Frankle and Carbin provided
empirical evidence that, at initialization, there exists a sub-graph of the
original deep learning model such that, when trained in isolation, it can
match the performance of the complete model Frankle and Carbin (2019). Indeed,
the authors claim that the parameters in the sub-graph have “won at the
lottery of initialization”. From a very practical perspective, this means that
all the parameters not in the selected sub-graph are, de facto, pruned from
the model, not requiring any gradient computation. This empirical evidence
potentially opens the road to the development of algorithms for pruning at
initialization. Unfortunately, at such point, two obstacles are yet to be
tackled. From one side, Frankle and Carbin (2019) uses an iterative procedure
to identify the sub-graph: it simply proves its existence but does not provide
a method to find it directly at initialization. On another side, it focuses on
un-structured sub-graphs, meaning that parameters are treated in isolation: is
it possible to find structured sub-graphs (or in other words, removing entire
nodes and not just arcs)?
Beyond the lottery ticket hypothesis. Many works of the last two years have
received inspiration from the lottery ticket hypothesis, in the quest of
determining the early lottery tickets. This is, however, a tough task to
solve: Frankle et al. (2020) shows that there is a region, at the very early
stages of learning, where the lottery tickets identified with iterative
pruning are “not stable” (meaning that tickets extracted at different moments
of this early stages are essentially different). This suggests that, in the
very first epochs, the neural network evolves in very different states, making
the problem of a-priori identifying winning tickets hard Tartaglione (2022).
This is also endorsed by other works, including Morcos et al. (2019); Malach
et al. (2020). On the other hand, other approaches reduce the overall
complexity of the iterative training by drawing early-bird tickets You et al.
(2019) (meaning that they learn the lottery tickets when the model has not yet
reached full convergence), even reducing the training data Zhang et al. (2021)
or moving the first steps towards structurally-sparse winning tickets, yet
still at iterative fashion, applying similar concept as Frankle and Carbin to
entire neurons and channels Chen et al. (2022).
With NEq we learn the important lessons related to the lottery ticket
hypothesis, targeting the reduction of computational complexity at training
time, without exploiting knowledge on pre-training models, or rewinding.
Hence, we do not target the achievement of sparse architectures, but we aim at
determining when a whole neuron requires to be updated or when the computation
of the gradient for its parameters is not necessary. As such, the comparison
with the other presented approaches will be in general unfair, as they require
a much greater computational complexity for training because of un-structured
sparsity (which introduces an overhead in the representation of the tensors)
and the iterative strategies. Furthermore, we will observe the possibility,
along the training process, that some neurons, already kept in a “frozen”
state, might unfreeze, requiring additional update steps. Although resource
re-allocation has been exploited before Tresp et al. (1996); He et al. (2018),
our unfreezing is different as it involves learning of a specific target
function by the neuron, and not learning new ones through their re-allocation.
In the next section, the notion of neuronal equilibrium will be presented, as
well as the strategy to determine which neurons will require gradient
computation.

##  3 Neurons at equilibrium

Figure 1: For a given time $t$ the model (either in blue or orange) receives
samples from the validation set (in red or green). The output of the $i$-th
neuron (whose cardinality is $N_{i}$) depends on both the model’s parameters
and the specific sample on the validation set. These outputs are squeezed,
concatenated and the obtained vector (of size $N_{i}\cdot\|\Xi_{val}\|_{0}$,
being $\|\Xi_{val}\|_{0}$ the cardinality of the validation set) is then
normalized, obtaining $\boldsymbol{\hat{y}}_{i}^{t}$.

In this section, we will treat the problem of determining when a given neuron,
along with the learning dynamics, finds itself at equilibrium. Towards this
end, we define $\boldsymbol{y}_{i,\xi}^{t}$ as the output of the $i$-th neuron
when the input $\xi$ is fed to the whole model trained after $t$ epochs. Given
a set of inputs $\xi\in\Xi_{val}$ (where $\Xi_{val}$ is the validation set),
it is possible to compare each $n$-th element $y_{i,n,\xi}^{t}$ with
$y_{i,n,\xi}^{t-1}$, for the same model’s input: what changes are the
parameters of the model. Fig. 1 provides an overview of the nomenclature used:
in the rest of the section we will see how to determine when a neuron is at
equilibrium.

###  3.1 Neuronal equilibrium

In this section, we are interested in evaluating when the relationship between
the input of the model and the output of the $i$-th neuron is modified. When
this happens, the neuron is at _non-equilibrium_ , meaning that its learned
function, in the whole picture (or in other words, taking into account the
evolution of the neurons in the previous layers as well), is still “evolving”.
We are interested in identifying the scenarios where the neuron is at
_equilibrium_ at the net of the interactions with the other neurons. To assess
it, let us define the cosine similarity between all the outputs of the $i$-th
neuron at time $t$ and at time $t-1$ for the whole validation set $\Xi_{val}$
as

$\phi_{i}^{t}=\sum_{\xi\in\Xi_{val}}\sum_{n=1}^{N_{i}}\hat{y}_{i,n,\xi}^{t}\cdot\hat{y}_{i,n,\xi}^{t-1}.$
(1)

Here we can determine that, when $\phi_{i}=1$, the $i$-th neuron produces the
same (eventually scaled) output between the evaluation at time $t$ and at time
$t-1$ for the same input $\xi$ of the model. We say the $i$-th neuron reaches
equilibrium when we have

$\lim_{t\rightarrow\infty}\phi_{i}^{t}=k,$ (2)

where $k\in[-1;+1]$ is some constant value. We can have the following
scenarios:

  * •

$k=1$. In this case, the two outputs are perfectly correlated, meaning that
the relationship bounding the input of the whole model $\xi$ and the output of
the specific $i$-th neuron is maintained.

  * •

$k\in(0;1)$. The outputs correlate, but we are in presence of an oscillatory
behavior (in the sense that the cosine similarity varies by a constant value
between consecutive evaluations). This effect can be caused by stochastic
effects like high learning rate/regularization, small batch size, or a
combination of them.

  * •

$k\in[-1;0]$. Also in this case we are in the presence of oscillatory
behavior, but the outputs are anti-correlated or de-correlated.

Here follows the evaluation framework to determine the arrival to an
equilibrium state for the $i$-th neuron.

###  3.2 Neuron dynamics evaluation

To assess the convergence to equilibrium for (2), it is important to evaluate
the variation of the similarities $\phi_{i}^{t}$ over time. Towards this end,
let us introduce the _variation of similarities_

$\Delta\phi_{i}^{t}=\phi_{i}^{t}-\phi_{i}^{t-1}.$ (3)

According to the analysis in Sec. 3.1, in this case, we say we reach
equilibrium when $\Delta\phi_{i}^{t}\rightarrow 0$. Hence, it is useful to
keep track of the recent evolution over the similarity scores in the model:
towards this end, we can introduce the velocity of the similarity variations:

$v_{\Delta\phi_{i}}^{t}=\Delta\phi_{i}^{t}-\mu_{eq}v_{\Delta\phi_{i}}^{t-1},$
(4)

where $\mu_{eq}$ is the momentum coefficient. We can rewrite (4) making the
similarity scores explicit, obtaining

$v_{\Delta\phi_{i}}^{t}=\left\\{\begin{array}[]{ll}\phi_{i}^{t}+\sum\limits_{m=1}^{t}(-1)^{m}\left[(\mu_{eq})^{m-1}+(\mu_{eq})^{m}\right]\phi_{i}^{t-m}&\mu_{eq}\neq
0\\\ \phi_{i}^{t}-\phi_{i}^{t-1}&\mu_{eq}=0,\end{array}\right.$ (5)

where $(\cdot)^{m}$ indicates power of $m$. If we assume
$\phi^{t}\in[0;1]\forall t$ (which is the case of ReLU-activated neurons), in
order to prevent (5) from exploding, we need to set $\mu_{eq}\in[0;0.5]$. We
can extend this setup to any layer if we assume that the neurons in the
trained model will not reach equilibrium with anti-correlated outputs.

###  3.3 Selection of trainable neurons at non-equilibrium

To evaluate when a given neuron has reached equilibrium, exploiting (2), we
can say that the $i$-th neuron is at equilibrium when it can satisfy

$\left|v_{\Delta\phi}^{t}\right|<\varepsilon,~{}~{}~{}~{}~{}\varepsilon\geq
0.$ (6)

It is important to notice that, once (6) is satisfied for a certain $t$, in
case something changes in the learning dynamics (for example, the learning
rate is re-scaled), there may exist some $t^{\prime}>t$ such that the
constraint is not satisfied anymore. When this happens, it means that the
neuron is driven towards _new states_ , and is no anymore at equilibrium.
Hence, it requires to be updated again.

###  3.4 Overall training scheme

Figure 2: Overall training scheme. In orange is the standard training part and
in blue is the neuron equilibrium evaluation and selection stages (we name
this whole part NEq).

The overall training scheme is summarized in Fig. 2. The model is trained for
one epoch, after which neurons at equilibrium are identified. We split this
into two phases: in the first (neuron dynamics evaluation), the velocity of
the similarities is evaluated according to (4), while in the second (trainable
neurons at non-equilibrium) the hidden neurons at non-equilibrium, which will
be trained for the next epoch, are identified according to (6). For the first
epoch, all the neurons are considered at non-equilibrium by default. The
evaluation of neurons at equilibrium is agnostic to the general training
strategy, which can include arbitrary re-scaling for the learning rate/hyper-
parameters or the most common optimizers. In the next section, we will test
this procedure on very different architectures, tasks, and learning
strategies.

##  4 Experiments

In this section, we report the experiments supporting the approach as
presented in Sec. 3.4. First, we will perform an ablation study, analyzing
single contributions for the introduced hyper-parameters and providing an
overview of neuronal equilibrium along the training process; then, we will
test the proposed technique on state-of-the-art network architectures,
datasets, and learning policies. All experiments were performed using 8 NVIDIA
A40 GPUs and the source code uses PyTorch 1.10.111the source code is available
at https://github.com/EIDOSLAB/NEq.

###  4.1 Ablation study

We performed our ablation study training a ResNet-32 He et al. (2015) model on
CIFAR-10 Krizhevsky et al. . Unless differently specified, following the
hyper-parameters setup of Zagoruyko and Komodakis (2016), the model is trained
using SGD as an optimizer, with a starting learning rate $\eta=0.1$ and
momentum $\mu_{opt}=0.9$ and a weight decay of $5\times 10^{-4}$ for 250
epochs. The learning rate is decayed by a factor of 10 after 100 and 150
epochs, using the formulation as in (4), with $\|\Xi_{val}\|_{0}=50$,
$\mu_{eq}=0.5$, and $\varepsilon=0.001$.

(a) ResNet-32 trained with SGD.

(b) ResNet-32 trained with Adam.

Figure 3: Back-propagation FLOPs (left, orange), updated neurons (center,
green), and classification accuracy (right, red) for ResNet-32 trained on
CIFAR-10.

####  4.1.1 SGD vs Adam

To show that our technique automatically self-adapts to the training policy,
we compare the evolution of the FLOPs required for a back-propagation step and
the number of updated neurons of two different training of the ResNet-32: one
using the SGD optimizer with $\mu_{opt}=0.9$, and the other using the Adam
optimizer. For Adam we leave the hyper-parameters to their default values
($\eta=0.001$, $\beta_{1}=0.9$, and $\beta_{2}=0.999$) and use the same weight
decay as for SGD ($5\times 10^{-4}$). Fig. 3 shows the trends for the two
training procedures. We can see that in the first phase of the train, where
$\eta$ is high, the amount of the trained neurons (and the FLOPs required for
the backward pass) is higher. This is related to the general lack of
equilibrium in the neurons of the network: at high learning rates, the
configuration of the neurons’ parameters is subjected to high stochastic
noise. As the train progresses, and the network progresses toward its final
configuration, fewer and fewer neurons need to be updated. Noticeably, Adam
drives the neurons towards equilibrium in a faster way, as expected; however,
in simple tasks like the considered one, converges to lower accuracy scores
($92.96\%$ for SGD and 92.01% for Adam). Furthermore, at the first learning
rate decay (epoch $100$), for SGD the number of updated neurons first
decreases and then increases, such a phenomenon is not present in the Adam
case. This is explained by the different working principles of the two
optimizers: SGD explores the solution space looking for large minima,
searching for configurations that prevent equilibrium in high learning rate
regimes.

Figure 4: Back-propagation FLOPs (left) and accuracy (right) for different
values of $\mu_{eq}$ for ResNet-32 trained on CIFAR-10.

(a) Distribution of $\phi$.

(b) Distribution of $\Delta\phi$.

(c) Distribution of $v_{\Delta\phi}$ with $\mu_{eq}=0.5$.

(d) Distribution of $v_{\Delta\phi}$ with $\mu_{eq}=0.9$.

Figure 5: Neuronal equilibrium-related quantities for ResNet-32 trained on
CIFAR-10. The red line indicates the average.

####  4.1.2 Distribution of $\phi$ & choice of $\mu_{eq}$

Looking at different values for $\mu_{eq}$ in Fig. 4, we observe for all the
values a convergence to similar accuracy. Despite without warranty from the
theory, we tested a very large value for the momentum coefficient ($0.9$): the
convergence of $v_{\Delta\phi_{i}}^{t}$ shows that the neurons are in general
in a very correlated case of equilibrium, with very high values for $k$ in
(2), which is also empirically observed in Fig. 5(a). However, including a
very large value for $\mu_{eq}$ maintains the memory of very old variations,
producing a sub-optimal reduction in terms of FLOPs reduction. We find that a
good compromise, supported by the findings as in Sec. 3.2, is to set
$\mu_{eq}$ to $0.5$. Fig. 5 reports the distribution for the velocities for
$\phi$, $\Delta\phi$, and $v_{\Delta\phi}$, observing that the average
converges to a specific $k$ for each of the three learning rates used.

Table 1: Ablation for ResNet-32 trained on CIFAR-10.

(a) Ablation on $\|\Xi_{val}\|_{0}$. $\boldsymbol{\|\Xi_{val}\|_{0}}$ | Bprop. FLOPs per iteration | Accuracy
---|---|---
500 |  84.73M $\pm$ 629.15K |  92.70 $\pm$ 0.12
250 |  82.62M $\pm$ 613.14K |  92.80 $\pm$ 0.43
100 |  84.82M $\pm$ 628.05K |  92.81 $\pm$ 0.15
50 |  84.81M $\pm$ 629.12K |  92.96 $\pm$ 0.21
25 |  84.49M $\pm$ 629.37K |  92.62 $\pm$ 0.28
10 |  84.91M $\pm$ 627.11K |  92.70 $\pm$ 0.27
5 |  84.34M $\pm$ 619.37K |  92.57 $\pm$ 0.48
2 |  85.76M $\pm$ 617.11K |  92.80 $\pm$ 0.24
1 |  85.56M $\pm$ 626.09K |  92.77 $\pm$ 0.23

(b) Ablation on $\varepsilon$. $\boldsymbol{\varepsilon}$ | Bprop. FLOPs per iteration | Accuracy
---|---|---
0.0 |  136.71M $\pm$ 15.34K |  92.62 $\pm$ 0.23
0.0001 |  124.45M $\pm$ 161.76K |  92.65 $\pm$ 0.40
0.0005 |  89.29M $\pm$ 589.71K |  92.69 $\pm$ 0.19
0.001 |  83.62M $\pm$ 629.22K |  92.96 $\pm$ 0.21
0.005 |  65.65M $\pm$ 591.07K |  91.72 $\pm$ 0.37
0.01 |  52.53M $\pm$ 590.30K |  91.23 $\pm$ 0.32
0.05 |  16.10M $\pm$ 254.94K |  86.80 $\pm$ 0.29
0.1 |  4.97M $\pm$ 186.54K |  83.90 $\pm$ 0.66
0.5 |  1.78M $\pm$ 137.92K |  76.78 $\pm$ 2.57

####  4.1.3 Impact of the validation set size and $\boldsymbol{\varepsilon}$

Tab. 1(b) provides an empirical evaluation of the impact on the performance
and on the FLOPs varying the validation set size. We indeed observe not a
significant impact on the performance of the model varying it. Interestingly,
the approach produces extremely good results even for extremely low
cardinality for the validation set (down to even a single image): this can be
explained by the presence of convolutional layers (the only fully-connected
layer is the output layer, excluded by default) which even with little images
produce high-dimensionality output in every neuron (Fig. 1) and by the
homogeneity of the considered dataset. Investigating the impact of
$\varepsilon$, instead, we find for very high values of $\varepsilon$ a drop
in performance, identifying a good compromise for classification tasks to
$0.001$.

###  4.2 Main experiments

Table 2: Results of the application of NEq to each experimental setup,
compared to the stochastic approach. We report the average FLOPs per iteration
at backpropagation and the final performance of the model evaluated on the
test set (values annotated with † report the classification accuracy, values
annotated with ‡ report the mean IoU).

Dataset | Model | Approach | Bprop. FLOPs per iteration | Performance
---|---|---|---|---
CIFAR-10 | ResNet-32 | Baseline | 138.94M $\pm$ 0.0M | 92.85% $\pm$ 0.23%†
Stochastic ($p=0.2$) | 112.99M $\pm$ 0.00M (-18.68%) | 92.78% $\pm$ 0.19% (-0.07%)†
Stochastic ($p=0.5$) | 69.75M $\pm$ 0.00M (-49.8%) | 91.88% $\pm$ 0.27% (-0.97%)†
Stochastic∗ | 86.34M $\pm$ 0.00M (-37.85%) | 92.23% $\pm$ 0.25% (-0.62%)†
Neq | 84.81M $\pm$ 0.63M (-38.96%) | 92.96% $\pm$ 0.21% (+0.11%)†
ImageNet-1K | ResNet-18 | Baseline | 3.64G $\pm$ 0.0G | 69.90% $\pm$ 0.04%†
Stochastic ($p=0.2$) | 2.94G $\pm$ 0.00G (-19.26%) | 69.42% $\pm$ 0.16% (-0.48%)†
Stochastic ($p=0.5$) | 1.85G $\pm$ 0.00G (-49.11%) | 69.18% $\pm$ 0.03% (-0.72%)†
Stochastic∗ | 2.82G $\pm$ 0.00G (-22.58%) | 69.45% $\pm$ 0.06% (-0.45%)†
Neq | 2.80G $\pm$ 0.03G (-23.08%) | 69.62% $\pm$ 0.06% (-0.28%)†
Swin-B | Baseline | 30.28G $\pm$ 0.00G | 84.71% $\pm$ 0.04% †
Stochastic ($p=0.2$) | 24.65G $\pm$ 0.00G (-18.6%) | 84.54% $\pm$ 0.04% (-0.83%)†
Stochastic ($p=0.5$) | 16.15G $\pm$ 0.00G (-46.67%) | 84.40% $\pm$ 0.02% (-0.31%)†
Stochastic∗ | 11.02G $\pm$ 0.00G (-63.67%) | 84.27% $\pm$ 0.04% (-0.44%)†
Neq | 10.78G $\pm$ 0.02G (-64.39%) | 84.35%$\pm$0.02% (-0.36%)†
COCO | DeepLabv3 | Baseline | 305.06G $\pm$ 0.0G | 67.71% $\pm$ 0.02%‡
Stochastic ($p=0.2$) | 248.69G $\pm$ 0.00G (-18.48%) | 67.11% $\pm$ 0.02% (-0.60%)‡
Stochastic ($p=0.5$) | 163.42G $\pm$ 0.00G (-46.43%) | 66.91% $\pm$ 0.04% (-0.80%)‡
Stochastic∗ | 229.00G $\pm$ 0.00G (-24.93%) | 67.02% $\pm$ 0.03% (-0.69%)‡
Neq | 217.29G $\pm$ 0.04G (-28.77%) | 67.22% $\pm$ 0.04% (-0.49%)‡

In this section, we show the results of the proposed method. For our
experiments, we focused on different state-of-the-art architectures trained on
standard classification and semantic segmentation datasets. All the learning
policies used are borrowed from other works and are un-optimized to test the
adaptability of NEq.
ResNet-32 trained on CIFAR-10. The training spans 250 epochs, using SGD as
optimizer with momentum $\mu_{opt}=0.9$, weight decay $5\times 10^{-4}$ and
initial learning rate $\eta=0.1$, reduced by a factor of $10$ after $100$ and
$150$ epochs. To evaluate the neuronal equilibrium we used a
$\|\Xi_{val}\|_{0}$ of $50$ images, $\mu_{eq}=0.5$, and $\varepsilon=0.001$.
We used a batch size of $100$ images during training.
ResNet-18 trained on ImageNet-1K Krizhevsky et al. (2012). This model was
trained with SGD as optimizer for 90 epochs, with $\eta=0.1$, reduced by a
factor of $10$ every $30$ epochs, $\mu_{opt}=0.9$ and weight decay $10^{-4}$
using a batch size of $128$. We used a $\|\Xi_{val}\|_{0}$ of 1.2k images,
$\mu_{eq}=0.5$, and $\varepsilon=0.001$.
Swin Transformer Liu et al. (2021) (Swin-B) trained on ImageNet-1K. To test
our technique on more modern models and training policies, we used the Swin-B
architecture. Here we trained the model starting from a pre-trained checkpoint
trained on ImageNet-21K, following the official GitHub
repository222https://github.com/microsoft/Swin-Transformer released under the
MIT License. We used a $\|\Xi_{val}\|_{0}$ of 1.2k images, $\mu_{eq}=0.5$, and
$\varepsilon=0.001$.
DeepLabv3 Chen et al. (2017) trained on COCO Lin et al. (2014). Other than
classification tasks of varying complexity, we tested our procedure on a
semantic segmentation problem. For this experiment, we used DeepLabv3 with a
ResNet-50 backbone and the COCO dataset. To train the network, we followed the
state-of-the-art procedure defined in
PyTorch333https://github.com/pytorch/vision/tree/main/references/segmentation.
We evaluated the neuronal equilibrium using a $\|\Xi_{val}\|_{0}$ of $320$
images, $\mu_{eq}=0.5$, and $\varepsilon=0.02$.

###  4.3 Discussion

The results (average over five different runs) are reported in Tab. 2. For
each experiment, we compare our technique with a “stochastic” approach.
Namely, we randomly halt, at every epoch and with probability $p$, the update
of a given neuron. We test on three different probabilities: $0.2$, $0.5$, and
a probability that is as close as possible to the average over the one
achieved by NEq - indicated with “*”. To evaluate the effectiveness of the
proposed procedure we focus on the average computational complexity of the
back-propagation for a single update iteration (expressed in FLOPs) and the
network generalization capabilities at the end of the training. In all the
considered scenarios, it is possible to observe a reduction of FLOPs with very
marginal or no performance drop for NEq. When compared to the stochastic
approach, with fixed probabilities, the amount of saved computation is similar
in all the considered scenarios, but the loss in performance varies, depending
on the specific architecture/dataset. On the contrary, NEq remains consistent
in performance, self-adapting to the specific setup and saving the largest
FLOPs for the given performance. Furthermore, testing the stochastic approach
with the same FLOPs saving (hence, even letting that information leak in favor
of the stochastic approach), the performance loss is lower.
Limitations. The current approach analyzes the behavior of an entire neuron.
However, empirical experiments show that there could be further improvements
considering ensembles of neurons. For example, Fig. 5 shows the average value
for the similarities close to a constant but many neurons are still away from
the convergence value, meaning that these neurons, at isolation, are still not
at equilibrium: is the scenario changing when considering the dynamics of
groups of neurons? Furthermore, to validate the adaptability of NEq to the
most popular training schemes, no optimization of the hyper-parameters for the
training procedure has been performed (as it is out of scope for our
evaluation). However, higher savings in computational complexity are possible
by tuning the training strategy as well. In such a direction, prospectively,
it will be of interest to design more efficient learning strategies which keep
into account the concept of neuronal equilibrium.

##  5 Conclusions

The work by Frankle and Carbin (2019) showed the existence of sub-graphs in
deep models which, when trained in isolation, can match the original
performance of the whole model. Finding these sub-graphs is, however, a
complex task, as in the first stages of the learning the model itself is at
non-equilibrium. Identifying these with a dynamic strategy, without requiring
a posterior over the whole training process, is a crucial task to be solved,
towards computational resources saving. Differently from the vast majority of
the literature which focuses on the identification of sub-graphs without any
concrete computational saving (as they rely on iterative or roll-back
algorithms), we have introduced the knowledge of neuronal equilibrium, looking
for entire structures of the deep model at equilibrium, not requiring further
optimization and gradient computation, which self-adapts to very specific
experimental setups on very different learning scenarios. This work opens the
doors toward a deeper understanding of the deep neural network’s learning
dynamics and to the development of new training strategies exploiting this
knowledge.

## Acknowledgements

This research was produced within the framework of Energy4Climate
Interdisciplinary Center (E4C) of IP Paris and Ecole des Ponts ParisTech. This
research was supported by 3rd Programme d’Investissements d’Avenir
[ANR-18-EUR-0006-02].

## References

  * Baydin et al. (2018) Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind.  Automatic differentiation in machine learning: a survey.  _Journal of Marchine Learning Research_ , 18:1–43, 2018\.
  * Blalock et al. (2020) Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag.  What is the state of neural network pruning?  _Proceedings of machine learning and systems_ , 2:129–146, 2020.
  * Brutzkus et al. (2018) A. Brutzkus, A. Globerson, E. Malach, and S. Shalev-Shwartz.  Sgd learns over-parameterized networks that provably generalize on linearly separable data.  2018\.
  * Chen et al. (2017) Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_ , 2017.
  * Chen et al. (2022) Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang.  Coarsening the granularity: Towards structurally sparse lottery tickets.  _arXiv preprint arXiv:2202.04736_ , 2022.
  * Evci et al. (2019) Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen.  The difficulty of training sparse neural networks.  _arXiv preprint arXiv:1906.10732_ , 2019.
  * Frankle and Carbin (2019) J. Frankle and M. Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  2019\.
  * Frankle et al. (2020) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin.  Linear mode connectivity and the lottery ticket hypothesis.  In _International Conference on Machine Learning_ , pages 3259–3269. PMLR, 2020.
  * Frankle et al. (2021) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin.  Pruning neural networks at initialization: Why are we missing the mark?  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=Ig-VyQc-MLK.
  * Gomez et al. (2019) Aidan N Gomez, Ivan Zhang, Siddhartha Rao Kamalakara, Divyam Madaan, Kevin Swersky, Yarin Gal, and Geoffrey E Hinton.  Learning sparse networks using targeted dropout.  _arXiv preprint arXiv:1905.13678_ , 2019.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385, 2015.
  * He et al. (2018) Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang.  Soft filter pruning for accelerating deep convolutional neural networks.  _arXiv preprint arXiv:1808.06866_ , 2018.
  * Jin et al. (2022) Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanumante, Zhengang Li, Zhiyu Chen, Yanzhi Wang, Kaiyuan Yang, and Sergey Tulyakov.  F8net: Fixed-point 8-bit only multiplication for network quantization.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=_CfpJazzXT2.
  * (14) Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pages 1097–1105, 2012.
  * LeCun et al. (1990) Yann LeCun, John S Denker, and Sara A Solla.  Optimal brain damage.  In _Advances in neural information processing systems_ , pages 598–605, 1990.
  * Lee et al. (2019) N. Lee, Thalaiyasingam Ajanthan, and P. Torr.  Snip: Single-shot network pruning based on connection sensitivity.  _ArXiv_ , abs/1810.02340, 2019.
  * Lee et al. (2018) Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr.  Snip: Single-shot network pruning based on connection sensitivity.  _arXiv preprint arXiv:1810.02340_ , 2018.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Louizos et al. (2017) Christos Louizos, Max Welling, and Diederik P Kingma.  Learning sparse neural networks through $l\\_0$ regularization.  _arXiv preprint arXiv:1712.01312_ , 2017.
  * Malach et al. (2020) Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir.  Proving the lottery ticket hypothesis: Pruning is all you need.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 6682–6691. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/malach20a.html.
  * Mhaskar and Poggio (2016) Hrushikesh N Mhaskar and Tomaso Poggio.  Deep vs. shallow networks: An approximation theory perspective.  _Analysis and Applications_ , 14(06):829–848, 2016.
  * Molchanov et al. (2017) Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov.  Variational dropout sparsifies deep neural networks.  In _International Conference on Machine Learning_ , pages 2498–2507. PMLR, 2017.
  * Morcos et al. (2019) Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian.  One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers.  _Advances in neural information processing systems_ , 32, 2019.
  * Mozer and Smolensky (1989) Michael C Mozer and Paul Smolensky.  Skeletonization: A technique for trimming the fat from a network via relevance assessment.  In _Advances in neural information processing systems_ , pages 107–115, 1989.
  * Plaut et al. (1986) David C Plaut et al.  Experiments on learning by back propagation.  1986\.
  * Sanh et al. (2020) Victor Sanh, Thomas Wolf, and Alexander Rush.  Movement pruning: Adaptive sparsity by fine-tuning.  _Advances in Neural Information Processing Systems_ , 33:20378–20389, 2020.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1355.  URL https://aclanthology.org/P19-1355.
  * Tanaka et al. (2020) Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli.  Pruning neural networks without any data by iteratively conserving synaptic flow.  _Advances in Neural Information Processing Systems_ , 33:6377–6389, 2020.
  * Tang et al. (2020) Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu.  Scop: Scientific control for reliable neural network pruning.  _arXiv preprint arXiv:2010.10732_ , 2020.
  * Tartaglione (2022) Enzo Tartaglione.  The rise of the lottery heroes: why zero-shot pruning is hard.  _arXiv preprint arXiv:2202.12400_ , 2022.
  * Tartaglione et al. (2018) Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, and Gianluca Francini.  Learning sparse neural networks via sensitivity-driven regularization.  _Advances in neural information processing systems_ , 31, 2018.
  * Tartaglione et al. (2021) Enzo Tartaglione, Andrea Bragagnolo, Francesco Odierna, Attilio Fiandrotti, and Marco Grangetto.  Serene: Sensitivity-based regularization of neurons for structured sparsity in neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2021\.
  * Tartaglione et al. (2022) Enzo Tartaglione, Andrea Bragagnolo, Attilio Fiandrotti, and Marco Grangetto.  Loss-based sensitivity regularization: towards deep sparse neural networks.  _Neural Networks_ , 146:230–237, 2022.
  * Tresp et al. (1996) Volker Tresp, Ralph Neuneier, and Hans-Georg Zimmermann.  Early brain damage.  _Advances in neural information processing systems_ , 9, 1996.
  * Wang et al. (2020) Chaoqi Wang, Guodong Zhang, and Roger Grosse.  Picking winning tickets before training by preserving gradient flow.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SkgsACVKPH.
  * Xiao et al. (2019) Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran.  Autoprune: Automatic network pruning by regularizing auxiliary parameters.  _Advances in neural information processing systems_ , 32, 2019.
  * Yang et al. (2020) Li Yang, Zhezhi He, and Deliang Fan.  Harmonious coexistence of structured weight pruning and ternarization for deep neural networks.  In _AAAI_ , pages 6623–6630, 2020.
  * You et al. (2019) Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin.  Drawing early-bird tickets: Towards more efficient training of deep networks.  _arXiv preprint arXiv:1909.11957_ , 2019.
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, _Proceedings of the British Machine Vision Conference (BMVC)_ , pages 87.1–87.12. BMVA Press, September 2016.  ISBN 1-901725-59-6.  doi: 10.5244/C.30.87.  URL https://dx.doi.org/10.5244/C.30.87.
  * Zhang et al. (2021) Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang.  Efficient lottery ticket finding: Less data is more.  In _International Conference on Machine Learning_ , pages 12380–12390. PMLR, 2021.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] The theory is presented in Sec. 3
and a deeper understanding of the single term’s contribution is presented in
Sec. 4.1; also experiments on popular architectures trained with very
different learning policies are performed and presented in Sec. 4.2 and Sec.
4.3.

    2. (b)

Did you describe the limitations of your work? [Yes] Limitations are discussed
in the specific paragraph of Sec. 4.3.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]
The work shows that there are learning regimes where not all the neurons need
to be updated in order to achieve a target performance. Considering that this
approach is not entailed for specific applications, no negative societal
impact can be found.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [Yes]
All the assumptions for the $\mu_{eq}$ estimations are stated in Sec. 3.2.

    2. (b)

Did you include complete proofs of all theoretical results? [Yes] Proofs and
additional results can be found in the supplementary material.

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
Code is included in the supplementary material, as well as instructions on how
to use it.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] Please see Sec. 4.2. The selection of the hyper-
parameters adopted is taken from the referenced sources.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] In the whole ablation study in Sec. 4.1 all
the standard deviations/error bars are reported.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Beginning of
Sec. 4.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] All the
existing datasets, models and code used is presented in Sec. 4 and credit is
given to the respective creators.

    2. (b)

Did you mention the license of the assets? [Yes] Where open source code is
used in Sec. 4 the license is mentioned.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] The code including the proposed technique is included in the
supplementary material.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]


# Real-time Deep Video Deinterlacing

Haichao Zhu  1234-5678-9012 The Chinese University of Hong Kong
<EMAIL_ADDRESS>,  Xueting Liu  The Chinese University of Hong Kong
<EMAIL_ADDRESS>,  Xiangyu Mao  The Chinese University of Hong Kong
<EMAIL_ADDRESS>and  Tien-Tsin Wong  The Chinese University of Hong
Kong<EMAIL_ADDRESS>
###### Abstract.

Interlacing is a widely used technique, for television broadcast and video
recording, to double the perceived frame rate without increasing the
bandwidth. But it presents annoying visual artifacts, such as flickering and
silhouette “serration,” during the playback. Existing state-of-the-art
deinterlacing methods either ignore the temporal information to provide real-
time performance but lower visual quality, or estimate the motion for better
deinterlacing but with a trade-off of higher computational cost. In this
paper, we present the first and novel deep convolutional neural networks
(DCNNs) based method to deinterlace with high visual quality and real-time
performance. Unlike existing models for super-resolution problems which relies
on the translation-invariant assumption, our proposed DCNN model utilizes the
temporal information from both the odd and even half frames to reconstruct
only the missing scanlines, and retains the given odd and even scanlines for
producing the full deinterlaced frames. By further introducing a layer-
sharable architecture, our system can achieve real-time performance on a
single GPU. Experiments shows that our method outperforms all existing
methods, in terms of reconstruction accuracy and computational performance.

Video deinterlace, image interpolation, convolutional neural network, deep
learning

††journal: TOG††journalvolume: X††journalnumber: X††article: XX††journalyear:
X††publicationmonth: 8††copyright: none††doi: 0000001.0000001_2††ccs:
Computing methodologies Reconstruction††ccs: Computing methodologies Neural
networks Figure 1. (a) Input interlaced frames. (b) Deinterlaced results
generated by SRCNN (Dong et al., 2016) re-trained with our dataset. (c) Blown-
ups from (b) and (d) respectively. (d) Deinterlaced results generated by our
method. The classical super-resolution method SRCNN reconstruct each frame
based on a single field and has large information loss. It also follows the
conventional translation-invariant assumption which does not hold for the
deinterlacing problem. Therefore, it inevitably generates blurry edges and
artifacts, especially around sharp boundaries. In contrast, our method can
circumvent this issue and reconstruct frames with higher visual quality and
reconstruction accuracy.

##  1\. Introduction

Interlacing technique has been widely used in the past few decades for
television broadcast and video recording, in both analog and digital ways.
Instead of capturing all $N$ scanlines for each frame, only $N/2$ odd numbered
scanlines are captured for the current frame (Fig. 2(a), upper), and the other
$N/2$ even numbered scanlines are captured for the following frame (Fig. 2(a),
lower). It basically trades the frame resolution for the frame rate, in order
to double the perceived frame rate without increasing the bandwidth.
Unfortunately, since the two half frames are captured in different time
instances, there are significant visual artifacts such as line flickering and
“serration” on the silhouette of moving objects (Fig. 2(b)), when the odd and
even fields are interlaced displayed. The degree of “serration” depends on the
motion of objects and hence is spatially varying. This makes deinterlacing
(removal of interlacing artifacts) an ill-posed problem.

Many deinterlacing methods have been proposed to suppress the visual
artifacts. A typical approach is to reconstruct two full frames from the odd
and even half frames independently (Fig. 2(c)). However, the result is usually
unsatisfactory, due to the large information loss (50% loss) (Doyle, 1990;
Wang et al., 2012, 2013). Higher-quality reconstruction can be obtained by
first estimating object motion (Jeon et al., 2009; Mohammadi et al., 2012; Lee
and Lee, 2013). However, motion estimation from half interlacing frames are
not reliable, and also computationally expensive. Hence, they are seldomly
used in practice, let alone real-time applications.

In this paper, we propose the first deep convolutional neural networks (DCNNs)
method tailormade for the video deinterlacing problem. To our best knowledge,
no DCNN-based deinterlacing method exists. One may argue that existing DCNN-
based methods for interpolation or super-resolution (Mallat, 2016; Dong et
al., 2016) can be applied to reconstruct the full frames from the half frames,
in order to solve the deinterlacing problem. However, such naive approach
lacks of utilizing the temporal information between the odd and even half
frames, just like the existing intra-field deinterlacing methods (Doyle, 1990;
Wang et al., 2012). Moreover, this naive approach follows the conventional
translation-invariant assumption. That means, all pixels in the output full
frames are processed with the same set of convolutional filters, even though
half of the scanlines (odd/even numbered) actually exist in the input half
frames. Fig. 3(b) shows a full frame, reconstructed by the state-of-the-art
DCNN-based super-resolution method, SRCNN (Dong et al., 2016), exhibiting
obvious halo artifact. Instead of replacing the potentially error-contaminated
pixels from the convolutional filtering with the groundtruth pixels in the
input half frames and leading to visual artifacts (Fig. 3(c)), we argue that
we should only reconstruct the missing scanlines, and leave the pixels in the
original odd/even scanlines intact. All these motivate us to design a novel
DCNN model tailored for solving the deinterlacing problem.

Figure 2. (a) Two half fields are captured in two distinct time instances. (b)
The interlaced display exhibits obvious artifacts on the silhouette of moving
car. (c) Two full frames reconstructed from the two half frames independently
with an intra-field deinterlacing method ELA (Doyle, 1990).

In particular, our newly proposed DCNN architecture circumvents the
translation-invariant assumption and takes the temporal information into
consideration. Firstly, we only estimate the missing scanlines to avoid
modifying the groundtruth pixel values from the odd/even scanlines (input).
That is, the output of the neural network system are two half frames
containing only the missing scanlines. Unlike most existing methods which
ignore the temporal information between the odd and even frames, we
reconstruct each half output frame from both the odd and even frames. In other
words, our neural network system takes two original half frames as input and
outputs two missing half frames (complements).

Since we have two outputs, two neural networks are needed for training. We
further accelerate it by combining the lower-levels of two neural networks
(Bengio, 2012), as the input are the same and hence the lower-level
convolutional filters are sharable. With this improved network structure, we
can achieve real-time performance.

Figure 3. (a) An input interlaced frame. (b) Directly applying SRCNN to
deinterlacing introduces blurry and halo artifacts. (c) The visual artifacts
are worsen if we retain the pixels from the input odd/even scanlines. (d) Our
result.

To validate our method, we evaluate it over a rich variety of challenging
interlaced videos including live broadcast, legacy movies, and legacy
cartoons. Convincing and visually pleasant results are obtained in all
experiments (Fig. 1 & 3(d)). We also compare our method to existing
deinterlacing methods and DCNN-based models in both visual comparison and
quantitative measurements. All experiments confirm that our method not only
outperforms existing methods in terms of accuracy, but also speed performance.

##  2\. Related Work

Before introducing our method, we first review existing works related to
deinterlacing. They can be roughly classified into tailor-made deinterlacing
methods, traditional image resizing methods, and DCNN-based image restoration
approaches.

_Image/Video Deinterlacing_

Figure 4. The architecture of the proposed convolutional neural network.

Image/video deinterlacing is a classic vision problem. Existing methods can be
classified into two categories: intra-field deinterlacing (Doyle, 1990; Wang
et al., 2012, 2013) and inter-field deinterlacing (Jeon et al., 2009;
Mohammadi et al., 2012; Lee and Lee, 2013). Intra-field deinterlacing methods
reconstruct two full frames from the odd and even fields independently. Since
there is large information loss (half of the data is missing) during frame
reconstruction, the visual quality is usually less satisfying. To improve
visual quality, inter-field deinterlacing methods incorporate the temporal
information between multiple fields from neighboring frames during frame
reconstruction. Accurate motion compensation or motion estimation (Horn and
Schunck, 1981) is needed to achieve satisfactory quality. However, accurate
motion estimation is hard in general. In addition, motion estimation requires
high computational cost, and hence inter-field deinterlacing methods are
seldom used in practice, especially for applications requiring real-time
processing.

_Traditional Image Resizing_ Traditional image resizing methods can also be
used for deinterlacing by scaling up the height of each field. To scale up an
image, cubic (Mitchell and Netravali, 1988) and Lanczos interpolation (Duchon,
1979) are frequently used. While they work well for low-frequency components,
high-frequency components (e.g. edges) may be over-blurred. More advanced
image resizing methods, such as kernel regression (Takeda et al., 2007) and
bilateral filter (Hung and Siu, 2012) can improve the visual quality by
preserving more high-frequency components. However, these methods may still
introduce noise or artifacts if the vertical sampling rate is less than the
Nyquist rate. More critically, they only utlize a single field and ignore the
temporal information, and hence suffer the same problem as intra-deinterlacing
methods.

_DCNNs for Image Restoration_ In recent years, deep convolutional neural
networks (DCNNs) based methods have been proposed to solve many image
restoration problems. Xie et al. (2012) proposed a DCNN model for image
denosing and inpainting. This model recovers the values of corrupted pixels
(or missing pixels) by learning the mapping between corrupted and uncorrupted
patches. Dong et al. (2016) proposed to adopt DCNN for image super-resolution,
which greatly outperforms the state-of-the-art image super-resolution methods.
Gharbi et al. (2016) further proposed a DCNN model for joint demosaiking and
denosing. It infers the values of three color channels of each pixel from a
single noisy measurement.

It seems that we can simply re-train these state-of-the-art neural network
based methods for our deinterlacing purpose. However, our experiments show
that visual artifacts are still unavoidable, as these DCNNs generally follow
the conventional translation-invariant assumption and modify the values of all
pixels, even in the known odd/even scanlines. Using a larger training dataset
or deeper network structure may alleviate this problem, but the computational
cost is drastically increased and still there is no guarantee that the values
of the known pixels remain intact. Even if we fix the values of the known
pixels (Fig. 3(c)), the quality does not improve. In contrast, we propose a
novel DCNN tailored for deinterlacing. Our model only estimates the missing
pixels instead of the whole frame, and also take the temporal information into
account to improve visual quality.

##  3\. Overview

Given an input interlaced frame $\mathbf{I}$ (Fig. 4(a)), our goal of
deinterlacing is to reconstruct two full size original frames $\mathbf{X}_{t}$
and $\mathbf{X}_{t+1}$ from $\mathbf{I}$ (Fig. 4(d)). We denote the odd field
of $\mathbf{I}$ as $\mathbf{X}^{\text{odd}}_{t}$ (blue pixels in Fig. 4(a)),
and the even field of $\mathbf{I}$ as $\mathbf{X}^{\text{even}}_{t+1}$ (red
pixels in Fig. 4(a)). The superscripts, odd and even, denote the odd- or even-
numbered half frames. The subscripts, $t$ and $t+1$, denote the two fields are
captured at two different time instances. Our goal is to reconstruct two
missing half frames, $\mathbf{X}^{\text{even}}_{t}$ (light blue pixels in Fig.
4(c)) and $\mathbf{X}^{\text{odd}}_{t+1}$ (pink pixels in Fig. 4(c)). Note
that we retain the known fields $\mathbf{X}^{\text{odd}}_{t}$ (blue pixels)
and $\mathbf{X}^{\text{even}}_{t+1}$ (red pixels) in our two output full
frames (Fig. 4(d)).

To estimate the unknown pixels $\mathbf{X}^{\text{even}}_{t}$ and
$\mathbf{X}^{\text{odd}}_{t+1}$ from the interlaced frame $\mathbf{I}$, we
propose a novel DCNN model (Fig. 4(b) & (c)). The input interlaced frame can
be of any resolution, and two half output images are obtained with five
convolutional layers. The weights of the convolutional operators are trained
from a DCNN model training procedure based on a prepared training dataset.
During the training phase, we synthesize a set of interlaced videos from
progressive videos of different types as the training pairs. The reason that
we need to synthesize interlaced videos for training is that no groundtruth
exists for the existing interlaced videos captured by interlaced scan devices.
The details of preparing the training dataset and the design of the proposed
DCNN are described in Section 4.

##  4\. DCNN-based Video Deinterlacing

###  4.1. Training Data Preparation

While there exists a large collection of interlaced videos over the Internet,
unfortunately, the ground-truth of these videos is lacking. Therefore, to
prepare a training data set, we have to synthesize interlaced videos from
existing progressive videos. To enrich our data variety, we collect $33$
videos from the Internet and capture $18$ videos using progressive scan
devices ourselves. The videos are of different genres, ranging from scenic,
sports, computer-rendered, to classic movies and cartoons. Then we randomly
sample $3$ pairs of consecutive frames from each collected video and obtain
$153$ frame pairs in total. For each pair of consecutive frames, we rescale
each frame to the size of $512\times 512$ and label them as the pair of
original frames $\mathbf{X}_{t}$ and $\mathbf{X}_{t+1}$ (ground-truth full
frames) (Fig. 5(a)). Then we synthesize an interlaced frame based on these two
original frames as
$\mathbf{I}=\\{\mathbf{X}^{\text{odd}}_{t},\mathbf{X}^{\text{even}}_{t+1}\\}$,
i.e., the odd lines of $\mathbf{I}$ are copied from $\mathbf{X}_{t}$ and the
even lines of $\mathbf{I}$ are copied from $\mathbf{X}_{t+1}$ (Fig. 5(b) & 6).
For each triplet
$\left\langle\mathbf{I},\mathbf{X}_{t},\mathbf{X}_{t+1}\right\rangle$ of
$512\times 512$ resolution, we further divide them into $64\times
64$-resolution patch triplets
$\left\langle\mathbf{I}_{p},\mathbf{X}_{t,p},\mathbf{X}_{t+1,p}\right\rangle$
with the sampling stride setting to $64$. Note that during patch generation,
the parity of the divided patches remain the same as original images. Finally,
for each patch triplet
$\left\langle\mathbf{I}_{p},\mathbf{X}_{t,p},\mathbf{X}_{t+1,p}\right\rangle$,
we use $\mathbf{I}_{p}$ as a training input (Fig. 5(b)) and the corresponding
$\mathbf{X}^{\text{even}}_{t,p}$ and $\mathbf{X}^{\text{odd}}_{t+1,p}$ as
training outputs (Fig. 5(c)). In particular, we convert patches into the Lab
color space and only use the L channel for training. Altogether, we collect
9,792 patch triplets from the prepared videos, where $80\%$ of the triplets
are used for training and the rest are used for validation during the training
process. Note that, although our model is trained by patches of $64\times 64$
resolution, the trained convolutional operators can actually be applied on
images of any resolution.

Figure 5. Training data preparation. (a) Two consecutive frames
$\mathbf{X}_{t}$ and $\mathbf{X}_{t+1}$ from an input video. (a) An interlaced
frame $\mathbf{I}$ is synthesized by taking the odd lines from
$\mathbf{X}_{t}$ and even lines from $\mathbf{X}_{t+1}$ respectively and
regarded as the training input. (c) The even lines of $\mathbf{X}_{t}$ and the
odd lines of $\mathbf{X}_{t+1}$ are regarded as the training output.
Figure 6. A real example of synthesizing an interlaced frame from two
consecutive progressive frames.

###  4.2. Neural Network Architecture

With the prepared training dataset, we now present how we design our network
structure for deinterlacing. An illustration of our network structure is shown
in Fig. 4. It contains five convolutional layers. Our goal is to reconstruct
the original two frames $\mathbf{X}_{t}$ and $\mathbf{X}_{t+1}$ from an input
interlaced frame $\mathbf{I}$. In the following, we first explain our design
rationales and then describe the architecture in detail.

Figure 7. Reconstructing two frames from two fields independently leads to
inevitable visual artifacts due to the large information loss.

_The Input/Output Layers_ One may suggest to utilize the existing neural
network (e.g. SRCNN (Dong et al., 2016)) to learn $\mathbf{X}_{t}$ from
$\mathbf{X}^{\text{odd}}_{t}$ and $\mathbf{X}_{t+1}$ from
$\mathbf{X}^{\text{even}}_{t+1}$ independently. This effectively turns the
problem into a super-resolution or image upscaling problem. However, there are
two drawbacks.

First of all, since the two frame reconstruction processes (i.e. from
$\mathbf{X}^{\text{odd}}_{t}$ to $\mathbf{X}_{t}$ and
$\mathbf{X}^{\text{even}}_{t+1}$ to $\mathbf{X}_{t+1}$) are independent from
each other, the neural network can only estimate the full frame from the known
half frame without the temporal information. This inevitably leads to less
satisfying results due to the large (50%) information loss. In fact, the two
fields in the interlaced frame are temporally correlated. Consider an extreme
case where the scene in the two consecutive frames are static. In this
scenario, the two consecutive frames are exactly the same, and the interlaced
frame should also be artifact-free and exactly equal to the groundtruth we are
looking for. However, using this naive super-resolution approach, we have to
feed the half frame $\mathbf{X}^{\text{odd}}_{t}$ (or
$\mathbf{X}^{\text{even}}_{t+1}$) to reconstruct a full frame. It completely
ignores the another half frame (which now contains the exact pixel values) and
introduces artifacts (due to 50% information loss). Fig. 7 shows the poor
result of one such scenario. In contrast, our proposed neural network takes
the whole interlaced frame $\mathbf{I}$ as input (Fig. 4(a)). Note that the
temporal information is implicitly taken into consideration in our network,
since the two fields captured at different time instances are used for
reconstructing each single frame. The network may exploit the temporal
correlation between fields to improve the visual quality in higher-level
convolutional layers.

Secondly, the standard neural network generally follows the conventional
translation-invariant assumption. That means all pixels in the input image are
processed with the same set of convolutional filters. However, in our
deinterlacing application, half of the pixels in $\mathbf{X}_{t}$ and
$\mathbf{X}_{t+1}$ actually exist in $\mathbf{I}$ and should be directly
copied from $\mathbf{I}$. Applying convolutional filters on these known pixels
inevitably changes their original colors and leads to clear artifacts (Fig.
3(b) & (c)). In contrast, our neural network only estimates the unknown pixels
$\mathbf{X}^{\text{even}}_{t}$ and $\mathbf{X}^{\text{odd}}_{t+1}$ (Fig. 4(c))
and copies the known pixels from $\mathbf{I}$ to $\mathbf{X}_{t}$ and
$\mathbf{X}_{t+1}$ directly (Fig. 4(d)).

_Pathway Design_ Since we estimate two half frames
$\mathbf{X}^{\text{even}}_{t}$ and $\mathbf{X}^{\text{odd}}_{t+1}$ from the
interlaced frame $\mathbf{I}$, we actually have to train two networks/pathways
independently. Separately training two networks is computational costly.
Instead of training two networks, one may suggest to train a single network
for estimating the two half frames simultaneously by doubling the depth of
each convolutional layer. However, this also highly increases the
computational cost, since the number of the trained weights are doubled. As
reported by (Bengio, 2012), deep neural network is to seek good representation
of input data, and such representations can be transferred to many other tasks
if the input data is similar. For example, the trained features of AlexNet
(Krizhevsky et al., 2012) (originally designed for object recognition) can
also be used for texture recognition and segmentation (Cimpoi et al., 2015).
In fact, the lower-level layers of the convolutional networks are always
lower-level feature detectors that can detect edges and other primitives.
These lower-level layers in the trained models can be reused for new tasks by
training new higher-level layers on top of them. Therefore, in our
deinterlacing scenario, it is natural to combine the lower-level convolutional
layers to reduce the computation, since the input of the two networks/pathways
is completely the same. On top of these weight-sharing lower-level layers,
higher-level layers are trained separately for estimating
$\mathbf{X}^{\text{even}}_{t}$ and $\mathbf{X}^{\text{odd}}_{t+1}$
respectively. This makes the higher-level layers more adaptable to different
objectives. Our method can be regarded as training one neural network for
estimating $\mathbf{X}^{\text{even}}_{t}$ and then fixing the first three
convolutional layers and re-training a second neural network for estimating
$\mathbf{X}^{\text{odd}}_{t+1}$.

_Detailed Architecture_ As illustrated in Fig. 4(b) & (c), our network
contains five convolutional layers with weights. The first, second, and third
layers are sequentially connected and shared by both pathways. The first
convolutional layer has $64$ kernels of size $3\times 3\times 1$. The second
convolutional layer has $64$ kernels of size $3\times 3\times 64$ and is
connected to the output of the first layer. The third convolutional layer has
$64$ kernels of size $1\times 1\times 64$ and is connected to the output of
the second layer. The forth and fifth layers branch into two pathways without
any connection between them. The forth convolutional layer has $64$ kernels of
size $3\times 3\times 64$ where each pathway has $32$ kernels. The fifth
convolutional has $2$ kernels of size $3\times 3\times 64$ where each pathway
has $1$ kernel. The activations for the first two layers are ReLU functions,
while for the rest layers are identify functions. The strides of convolution
for the first four layers are $1$ pixel. For the last layer, the horizontal
stride remains $1$ pixel, while the vertical stride is $2$ pixels to obtain
half-height images.

###  4.3. Learning and Optimization

Given the training dataset containing a set of triplets
$\left\langle\mathbf{I}_{p},\mathbf{X}^{\text{even}}_{t,p},\mathbf{X}^{\text{odd}}_{t+1,p}\right\rangle$,
the optimal weights $W^{*}$ of our neural network are trained via the
following objective function:

(1)
$\begin{split}W^{*}=\operatorname*{\arg\min}\frac{1}{N_{p}}&\sum_{p}\Big{(}\|\widehat{\mathbf{X}}^{\text{even}}_{t,p}-\mathbf{X}^{\text{even}}_{t,p}\|^{2}_{2}+\|\widehat{\mathbf{X}}^{\text{odd}}_{t+1,p}-\mathbf{X}^{\text{odd}}_{t+1,p}\|^{2}_{2}\\\
&+\lambda_{TV}\big{(}TV(\widehat{\mathbf{X}}_{t,p})+TV(\widehat{\mathbf{X}}_{t+1,p})\big{)}\Big{)}\end{split}$

where $N_{p}$ is the number of training samples,
$\widehat{\mathbf{X}}^{\text{even}}_{t,p}$ and
$\widehat{\mathbf{X}}^{\text{odd}}_{t+1,p}$ are the estimated output of the
neural network, $TV(\cdot)$ is the total variation regularizer (Aly and
Dubois, 2005; Johnson et al., 2016) and $\lambda_{TV}$ is the regularization
scalar.

We trained our neural network using Tensorflow on a workstation equipped with
a single nVidia TITAN X Maxwell GPU. The standard ADAM optimization method
(Kingma and Ba, 2014) is used to solve Eq. 1. The learning rate is $0.001$ and
$\lambda_{TV}$ is set to $2\times 10^{-8}$ in our experiments. The number of
epochs is $200$ and the batch size for each epoch is $64$. It takes about $4$
hours to train the neural network.

##  5\. Result and Discussion

Figure 8. Comparisons between bicubic interpolation, SRCNN (Dong et al., 2016)
and our method.
Figure 9. Comparisons between the state-of-the-art deinterlacing tailored
methods, including ELA (Doyle, 1990), WLSD (Wang et al., 2014), and FBA
(Vedadi and Shirani, 2013), with our method.

We evaluate our method on a large collection of interlaced videos downloaded
from the Internet or captured by ourselves with interlaced scan cameras. These
videos include live sporting videos (“Soccer” in Fig. 1 and “Tennis” in Fig.
8), scenic videos (“Leaves” in Fig. 1 and “Bus” in Fig. 8), computer-rendered
gameplay videos (“Hunter” in Fig. 8), legacy movies (“Haystack” in Fig. 8),
and legacy cartoons (“Rangers” in Fig. 8). Note that, we have no access to the
original progressive frames (groundtruth) of these videos. Without
groundtruth, we can only compare our method to existing methods visually, but
not quantitatively.

To evaluate quantitatively (with comparison to the groundtruth), we synthesize
a set of test interlaced videos from progressive scan videos of different
genres. None of these synthetic interlaced videos exist in our training data.
Fig. 9 presents a set of synthetic interlaced videos, including sports
(“Basketball”), scenic (“Taxi”), computer-rendered (“Roof”), movies
(“Jumping”), and cartoons (“Tide” and “Girl”). Due to the page limit, we only
present one representative interlaced frame for each video sequence. While two
full size frames can be recovered from each single interlaced frame, we only
show the first frame in all our results. Please refer to the supplementary
materials for more complete results.

_Visual Comparison_ We first compare our method with the classic bicubic
interpolation and the existing DCNN tailored for super-resolution, i.e. SRCNN
(Dong et al., 2016). Since SRCNN is not designed for deinterlacing, we re-
train their model with our prepared dataset for deinterlacing purpose. The
results are presented in Fig. 1 and 8. “Soccer”, “Bus” and “Tennis” are in
$1080i$ format and exhibit severe interlacing artifacts. Besides, the frames
also contain motion-blur and video compression artifacts. Since both bicubic
interpolation and SRCNN reconstruct each frame from a single field alone,
their results are unsatisfactory and exhibit obvious artifacts due to the
large information loss. SRCNN performs even worse than the bicubic
interpolation, since it follows the conventional translation-invariant
assumption which not held in deinterlacing scenario. In comparison, our method
can obtain much clearer and sharper results than our competitors. The “Hunter”
example shows a moving character from a gameplay where the computer-rendered
object contours/boundaries are sharply preserved. Both bicubic interpolation
and SRCNN lead to blurry and zig-zag near these sharp edges. In contrast, our
method obtains the best reconstruction result in achieving sharp and smooth
boundaries. The “Haystack” and “Rangers” examples are both taken from legacy
DVDs in interlaced NTSC format. In the “Haystack” example, only the character
is moving, while the background remains static. Without considering the
temporal information, both bicubic interpolation and SRCNN fails to recover
the fine texture of the haystacks and obtain blurry results. In sharp
contrast, our method successfully recovers the fine texture by taking two
fields into consideration.

PSNR/SSIM | Taxi | Roof | Basketball | Jumping | Tide | Girl
---|---|---|---|---|---|---
bicubic | 31.56/0.9453 | 33.11/0.9808 | 34.67/0.9783 | 37.81/0.9801 | 31.87/0.9809 | 29.14/0.9585
ELA | 32.47/0.9444 | 34.41/0.9839 | 32.08/0.9605 | 38.82/0.9844 | 33.89/0.9811 | 31.62/0.9724
WLSD | 35.99/0.9746 |  35.70/0.9883 | 35.05/0.9794 | 38.19/0.9819 | 34.17/0.9820 | 32.00/0.9761
FBA | 34.94/0.9389 | 35.26/0.9815 | 33.93/0.9749 | 38.27/0.9822 | 35.15/0.9822 | 31.78/0.9756
SRCNN | 30.12/0.9214 | 32.01/0.9749 | 29.18/0.9353 | 36.81/0.97094 | 33.02/0.9758 | 27.79/0.9477
Ours |  38.15/0.9834 | 35.44/0.9866 |  36.55/0.9838 |  39.75/0.9889 |  35.37/0.9807 |  35.44/0.9866
Table 1. PSNR and SSIM between the deinterlaced frames and groundtruth of all methods. Average time (s) | ELA | WLSD | FBA | Bicubic | SRCNN | Our Methods
---|---|---|---|---|---|---
With sharable layers | Without sharable layers
$1920\times 1080$ | 0.6854 | 2.9843 | 4.1486 | 0.7068 | 0.3010 | 0.0835 | 0.2520
$1024\times 768$ | 0.0676 | 1.0643 | 1.6347 | 0.2812 | 0.0998 | 0.0301 | 0.0833
$720\times 576$ | 0.0317 | 0.4934 | 0.6956 | 0.1176 | 0.0423 | 0.0204 | 0.0556
$720\times 480$ | 0.0241 | 0.4956 | 0.7096 | 0.1110 | 0.0419 | 0.0137 | 0.0403
Table 2. Timing statistics for all methods.

We further compare our method to the state-of-the-art deinterlacing methods,
including ELA (Doyle, 1990), WLSD (Wang et al., 2014), and FBA (Vedadi and
Shirani, 2013). ELA is the most widely used deinterlacing methods due to its
high performance. It is an intra-field method and uses edge directional
correlation to reconstruct the missing scanlines. WLSD is the state-of-the-art
intra-field deinterlacing method based on optimization. It generally produces
better result than that of ELA, but with a higher computational expense. FBA
is the state-of-the-art inter-field method. Fig. 9 shows the results of all
methods for a set of synthetic interlaced videos, in which we have the
groundtruths for quantitative evaluation. Besides the reconstructed frames, we
also blow-up the difference images for better visualization. The difference
image is simply computed as the pixel-wise absolute difference between the
output and the groundtruth. As we can observe, all our competitors generate
artifacts surrounding the boundaries. The sharper the boundary is, the more
obvious the artifact is. In general, ELA produces the most artifacts since it
adopts a simple interpolator and utilizes information from a single field
alone. WLSD produces less artifacts as it adopts a more complex optimization-
based strategy to fill the missing pixels. But it still only utilizes
information of a single field and has large information loss during
reconstruction. Though FBA utilizes the temporal information, it still cannot
achieve good visual quality because they only rely on simple interpolators. In
contrast, our method produces significantly less artifacts than all
competitors.

_Quantitative Evaluation_ We train our neural network by minimizing the loss
of Eq. 1 on the training data. The training loss and validation loss
throughout the whole training epochs are shown in Fig. 10. Both training and
validation losses reduce rapidly after the first few epochs and converge in
around 50 epochs.

We also compare the accuracy of our method to our competitors in terms of peak
signal-to-noise ratio (PSNR) and structural similarity index (SSIM). Note that
we only compute the PSNR and SSIM for those test videos with groundtruth. We
take the average value over all frames of each video sequence in computing
both measurements. Table 1 presents the statistics. Our method outperforms the
competitors in terms of both PSNR and SSIM in most cases.

_Timing Statistics_ Lastly, we compare the running time of our method to our
competitors on a workstation with Intel Core CPU i7-5930, 65GB RAM equipped
with a nVidia TITAN X Maxwell GPU. The statistics are presented in Table 2.
Our method achieves the highest performance among all methods in all
resolutions. It processes even faster than ELA with apparently better visual
quality. ELA and SRCNN have similar performance and are slighter slower than
our method. Bicubic interpolation, WLSD, and FBA have much higher
computational complexity and are far from real-time processing. Note that ELA
is only a CPU method without GPU acceleration. In particular, with a single
GPU, our method already achieves real-time performance up to the resolution of
$1024\times 768$ (33 fps). With one more GPU, our method can also achieve
real-time performance for $1920\times 1080$-resolution videos. We also test
our model without sharing lower-level layers, i.e., two separate networks are
needed for reconstructing the two frames. The statistics is shown in the last
column in Table 2. This strategy roughly triples the computational time while
quality is similar to that with sharing low-level layers.

Figure 10. Training loss and validation loss of our neural network.

_Limitations_ Since our method does not explicitly separate the two fields for
reconstructing two full frames, the two fields may interfere each other badly
when the motion between the two fields are extremely large. The first row in
Fig. 11 presents an example where the interlaced frame has a very large
motion, obvious artifacts can be observed. Our method may also fail when the
interlaced frame contains very thin horizontal structures. The second row of
Fig. 11 shows an example where a horizontal thin reflection stripe appears on
a car. Only one line of the reflection stripe is scanned in the interlaced
frame. Our neural network fails to identify it as a result of interlacing, but
regards it as the original structures and incorrectly preserves it in the
reconstructed frame. This is because this kind of patches is rare and gets
diluted by the large amount of common cases. We may relieve this problem by
training the neural network with more such training patches.

Figure 11. Failure cases. The top row shows a case where our result contains
obvious artifacts when the motion of the interlaced frame is too large. The
bottom row shows a case where our method fails to identify thin horizontal
structures as interlacing artifacts and incorrectly preserves it in the
reconstructed frame.

##  6\. Conclusion

In this paper, we present the first DCNN for video deinterlacing. Unlike the
conventional DCNNs suffering from the translation-invariant issue, we proposed
a novel DCNN architecture by adopting the whole interlaced frame as input and
two half frames as output. We also propose to share the lower-level
convolutional layers for reconstructing the two output frames to boost
efficiency. With this strategy, our method achieves real-time deinterlacing on
a single GPU for videos of resolution up to $1024\times 768$. Experiments show
that our method outperforms existing methods, including traditional
deinterlacing methods and DCNN-based models re-trained for deinterlacing, in
terms of both reconstruction accuracy and computational performance.

Since our method takes the whole interlaced frame as the input, frame
reconstruction is always influenced by both fields. While this may produce
better results in most of the cases, it occasionally leads to visually poorer
results when the motion between two fields is extremely large. In this
scenario, reconstructing each frame from a single field without considering
temporal information may produce better results. A possible solution is to
first recognize such large-motion frames, and then decide whether temporal
information should be utilized for deinterlacing.

## References

  * [1]
  * Aly and Dubois [2005] Hussein A. Aly and Eric Dubois. 2005.  Image up-sampling using total-variation regularization with a new observation model.  IEEE Transactions on Image Processing 14, 10 (2005), 1647–1659.
  * Bengio [2012] Yoshua Bengio. 2012\.  Deep learning of representations for unsupervised and transfer learning.  Proceedings of ICML Workshop on Unsupervised and Transfer Learning 27, 17–36.
  * Cimpoi et al. [2015] Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi. 2015\.  Deep filter banks for texture recognition and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3828–3836.
  * Dong et al. [2016] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016\.  Image super-resolution using deep convolutional networks.  IEEE transactions on pattern analysis and machine intelligence 38, 2 (2016), 295–307.
  * Doyle [1990] T. Doyle. 1990.  Interlaced to sequential conversion for EDTV applications. In Proceedings of International Workshop on Signal Processing of HDTV. 412–430.
  * Duchon [1979] Claude E. Duchon. 1979\.  Lanczos filtering in one and two dimensions.  Journal of Applied Meteorology 18, 8 (1979), 1016–1022.
  * Gharbi et al. [2016] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016.  Deep joint demosaicking and denoising.  ACM Transactions on Graphics 35, 6 (2016), 191\.
  * Horn and Schunck [1981] Berthold K.P. Horn and Brian G. Schunck. 1981.  Determining optical flow.  Artificial intelligence 17, 1-3 (1981), 185–203.
  * Hung and Siu [2012] K.W. Hung and W.C. Siu. 2012.  Fast image interpolation using the bilateral filter.  IET Image Processing 6, 7 (2012), 877–890.
  * Jeon et al. [2009] Gwanggil Jeon, Jongmin You, and Jechang Jeong. 2009\.  Weighted fuzzy reasoning scheme for interlaced to progressive conversion.  IEEE Transactions on Circuits and Systems for Video Technology 19, 6 (2009), 842–855.
  * Johnson et al. [2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016.  Perceptual losses for real-time style transfer and super-resolution. In Proceedings of European Conference on Computer Vision. Springer, 694–711.
  * Kingma and Ba [2014] Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 (2014).
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.
  * Lee and Lee [2013] Kwon Lee and Chulhee Lee. 2013.  High quality spatially registered vertical temporal filtering for deinterlacing.  IEEE Transactions on Consumer Electronics 59, 1 (2013), 182–190.
  * Mallat [2016] Stéphane Mallat. 2016\.  Understanding deep convolutional networks.  Philosophical Transactions of the Royal Society A 374, 2065 (2016), 20150203.
  * Mitchell and Netravali [1988] Don P. Mitchell and Arun N. Netravali. 1988.  Reconstruction filters in computer-graphics. In Computer Graphics. 221–228.
  * Mohammadi et al. [2012] H. Mahvash Mohammadi, Y. Savaria, and J.M.P. Langlois. 2012\.  Enhanced motion compensated deinterlacing algorithm.  IET Image Processing 6, 8 (2012), 1041–1048.
  * Takeda et al. [2007] Hiroyuki Takeda, Sina Farsiu, and Peyman Milanfar. 2007\.  Kernel regression for image processing and reconstruction.  IEEE Transactions on image processing 16, 2 (2007), 349–366.
  * Vedadi and Shirani [2013] Farhang Vedadi and Shahram Shirani. 2013.  De-Interlacing Using Nonlocal Costs and Markov-Chain-Based Estimation of Interpolation Methods.  IEEE Transactions on Image Processing 22, 4 (2013), 1559–1572.
  * Wang et al. [2012] Jin Wang, Gwanggil Jeon, and Jechang Jeong. 2012.  Efficient adaptive deinterlacing algorithm with awareness of closeness and similarity.  Optical Engineering 51, 1 (2012), 017003–1.
  * Wang et al. [2013] Jin Wang, Gwanggil Jeon, and Jechang Jeong. 2013.  Moving Least-Squares Method for Interlaced to Progressive Scanning Format Conversion.  IEEE Transactions on Circuits and Systems for Video Technology 23, 11 (2013), 1865–1872.
  * Wang et al. [2014] Jin Wang, Gwanggil Jeon, and Jechang Jeong. 2014.  De-Interlacing algorithm using weighted least squares.  IEEE Transactions on Circuits and Systems for Video Technology 24, 1 (2014), 39–48.
  * Xie et al. [2012] Junyuan Xie, Linli Xu, and Enhong Chen. 2012.  Image denoising and inpainting with deep neural networks. In Advances in Neural Information Processing Systems. 341–349.