# Neural Compression-Based Feature Learning for Video Restoration

Cong Huang${}^{1}\thanks{ This work was done when Cong Huang was an intern at
Microsoft Research Asia. }\qquad\\!$ Jiahao Li ${}^{2}\qquad\\!$ Bin
Li${}^{2}\qquad\\!$ Dong Liu${}^{1}\qquad\\!$ Yan Lu2
1 University of Science and Technology of China $\ \ {}^{2}$ Microsoft
Research Asia
<EMAIL_ADDRESS><EMAIL_ADDRESS>{li.jiahao, libin,
<EMAIL_ADDRESS>This work was done when Cong Huang was an intern at
Microsoft Research Asia.

###### Abstract

How to efficiently utilize the temporal features is crucial, yet challenging,
for video restoration. The temporal features usually contain various noisy and
uncorrelated information, and they may interfere with the restoration of the
current frame. This paper proposes learning noise-robust feature
representations to help video restoration. We are inspired by that the neural
codec is a natural denoiser. In neural codec, the noisy and uncorrelated
contents which are hard to predict but cost lots of bits are more inclined to
be discarded for bitrate saving. Therefore, we design a neural compression
module to filter the noise and keep the most useful information in features
for video restoration. To achieve robustness to noise, our compression module
adopts a spatial-channel-wise quantization mechanism to adaptively determine
the quantization step size for each position in the latent. Experiments show
that our method can significantly boost the performance on video denoising,
where we obtain 0.13 dB improvement over BasicVSR++ with only 0.23x FLOPs.
Meanwhile, our method also obtains SOTA results on video deraining and
dehazing.

##  1 Introduction

Figure 1:  (a) Previous framework without temporal feature refinement. (b) Our
framework via neural compression-based feature learning. (c) t-SNE [40]
visualization. $c^{gt}_{t}$ are sampled from clean video (hypersmooth, Set8
[39]). For $c_{t}$ and $\tilde{c}_{t}$, we add different additive white
Gaussian noises (same noise $\sigma$ but different noise random seeds) to the
same input video to sample these feature points. It shows that $\tilde{c}_{t}$
are more robust to noise and get closer to $c^{gt}_{t}$. (d) Performance
comparison on video denoising (Set8, noise $\sigma=50$).

Video restoration aims to recover the high-quality video from the degraded
input. Typical degradation includes various noises, rain, haze, etc. It has a
wide range of applications, but this problem is still under-explored.
Different from image restoration that focuses on the intrinsic proprieties in
single image [49], video restoration relies more on extracting and utilizing
temporal features for better quality.

Recent video restoration methods mainly focus on network structure design for
better extracting temporal features. For example, RViDeNet [48] and EDVR [41]
use deformable convolution to align the features of neighboring frames.
BasicVSR [8] designs a bi-directional feature propagation network. BasicVSR++
[9] introduces the second-order grid propagation network structure and flow-
guided deformable alignment network. However, these methods directly use the
extracted temporal features without any refinement. The temporal features
usually contain lots of noisy and irrelevant information, which interferes
with the restoration of the current frame. In this paper, we take video
denoising as a case study and explore how to utilize the extracted temporal
features efficiently.

We propose a novel neural compression-based solution to refine the features
and learn noise-robust feature representations. From the perspective of neural
codec, the noisy data usually contains lots of high-frequency and is hard to
predict. To save the bitrate, codec prefers to discard these noisy and
uncorrelated contents. This motivates us to design a neural compression module
to purify the temporal features and filter the noisy information therein for
video restoration. To achieve robustness to noise, namely let the
representations of the noise-perturbed data be mapped to the same quantized
representation with the clean data with high probability, the quantization
step needs to be properly set. However, most existing neural compression
frameworks only support fixed quantization step size. This cannot meet our
purpose and even harms the inherent textures. To solve this problem, we design
an adaptive quantization mechanism at spatial-channel-wise for our compression
module, where the quantization step is learned by our prior model. Our
quantization mechanism can adaptively purify the features with different
content characteristics. During the training, the cross-entropy loss is used
to guide the learning of the compression module and helps preserve the most
useful information.

Fig. 1 shows the framework comparison. From the t-SNE [40] visualization shown
in Fig. 1 (c), we find, via our neural compression-based feature learning, the
features are more robust to noise and get closer to the features generated
from the clean video. Fig. 1 (d) is the performance comparison. We observe
that, empowered by the noise-robust feature representations, our framework
significantly improves the restoration quality, when compared with prior
state-of-the-art (SOTA) methods. The major contributions of this paper are
summarized as follows:

$\bullet$ We propose a novel neural compression-based feature learning for
video restoration. After processed by our compression module, the features are
more robust to noise and then improve the restoration quality.

$\bullet$ To achieve robustness to noise and adaptively purify the features
with different content characteristics, we design a learnable quantization
mechanism at spatial-channel-wise.

$\bullet$ To further boost the performance, we also design an attention module
to help the feature learning, and a motion vector refinement module to improve
the discontinuous motion vector estimated from noisy video.

$\bullet$ We propose a lightweight framework. Compared with previous SOTA
methods, our method achieves a better quality-complexity trade-off on video
denoising, deraining, and dehazing.

Figure 2: Comparison between traidtional codec x265 [1] and neural codec [20]
when compressing the noised video (additive white Gaussian noise with
$\sigma=20$). BPP denotes bits per pixel, which measures bitrate cost.

##  2 Related Work

###  2.1 Video Restoration

Existing video restoration methods leveraging temporal correlation can be
divided into two categories: the sliding window-based methods and the
recurrent methods.

The sliding window-based methods take several adjacent frames as input for
each frame. Some methods [13, 39] do not depend on explicit motion alignments.
VNLNet [13] uses the non-local module to search similar patches across frames.
FastDVDNet [39] uses stacked U-Net [37] to progressively fuse the unaligned
neighboring frames. By contrast, ToFlow [44] and DVDNet [38] use a motion
estimation component to explicitly align neighboring frames. To explore more
temporal correlation, RViDeNet [48] and EDVR [41] propose feature domain
alignment. They align the features of neighboring frames rather than raw
pixels and this mechanism is adopted by most recent methods.

The sliding window-based methods suffer from a narrow temporal scope and
cannot leverage the information outside the sliding window. By contrast, the
recurrent methods learn the temporal features within a long temporal range,
and achieve better performance. EMVD [29] recurrently combines all past frames
as auxiliary information. Yan et al. [45] proposed a recurrent feature
propagation framework without explicit alignment. The feature propagation in
BasicVSR [8] uses the explicit alignment. Recently, BasicVSR++ [9] achieves
excellent performance by using a second-order grid propagation structure and a
flow-guided deformable alignment module.

###  2.2 Video Compression

Traditional video codecs, e.g, H.264 and H.265, adopt the hybrid framework
which consists of prediction, transform, quantization, entropy coding, and
loop-filter. Benefiting from the progress of neural image compression [4, 5,
31], neural video compression [28, 2, 23, 24, 20] recently also has a great
development. For example, Lu  et al. [28] designed the DVC model, which
follows the framework of traditional video codec but uses neural networks to
implement all modules therein. Following DVC, Agustsson  et al. [2] designed a
more advanced optical flow estimation in scale space. Recently Li  et al. [20]
proposed a conditional coding-based framework which achieves better
performance.

##  3 Motivation

Our motivation comes from that video compression can filter the noise. Video
compression aims at using the least bitrate cost to represent the video. For
traditional codec, the residuals of noisy contents are usually large as they
are hard to predict from reference frames. These residuals contains lots of
high-frequency and will consume many bits. To achieve the bitrate saving,
traditional codec uses the quantization to discard the residuals of noisy
contents, especially for the high-frequency therein, which is like a low-pass
filter. We use the traditional codec x265 [1] to conduct an analysis
experiment, as shown in Fig. 2. From Fig. 2 (c), we find that the traditional
codec x265 can filter the noise in a large degree. Fig. 2 (d) shows, when
allocated more bits, x265 will encode the noise but in a much smoother way.

Different from traditional codec using linear DCT (discrete cosine transform),
neural codec will learn a neural encoder to transform video from pixel domain
to latent feature domain. The latent feature is then quantized, and its
distribution is estimated to perform arithmetic coding. The distribution is
predicted more accurately, more bitrate saving is achieved. However, the
distributions of the noisy and uncorrelated contents are hard to predict well.
Thus, to save the bitrate, these contents are more inclined to be discarded
guided by the cross-entropy loss. Fig. 2 (e) shows the effectiveness of neural
codec [20] (model weights are provided by authors of [20]). In particular, the
neural codec can much better remove the noise therein and keep more semantic
information when compared with x265.

Inspired by this analysis, we propose utilizing a neural codec to help video
restoration. The neural codec is used to filter the noisy information in
features via the quantization. If the quantization step and data distribution
are properly learned, the representations of the noise-perturbed data will be
mapped to the same quantized representation with the clean data with high
probability. The noise-robust feature representations will improve the final
restoration quality.

Another advantage of using neural codec rather than traditional codec is that
the neural codec can be end-to-end trainable and will have better performance
when jointly trained with other restoration modules.

Figure 3: The overall framework of our method. The detailed network structures
of each module can be found in appendices.

##  4 Proposed Method

###  4.1 Framework Overview

We design a neural compression-based framework for video restoration. Our
framework contains three parts: feature alignment, feature refinement for
learning noise-robust feature representations, and feature fusion. The
framework is illustrated in Fig. 3.

Feature alignment. Given the noisy frames $x_{t-1}$ and $x_{t}$, we first use
motion estimation to estimate the motion vector (MV) $mv_{t}$. Then we design
an MV refinement module to improve the discontinuous MV $mv_{t}$ estimated
from noisy video. With the refined MV $\widetilde{mv}_{t}$, the coarse
features $\hat{c}_{t}$ are obtained via a bilinear warping function.

Feature refinement. As $\hat{c}_{t}$ contains some noisy and uncorrelated
information, we propose a neural compression-based feature refinement to
purify the features. It is noted that, our feature refinement part consists of
two modules. One is the attention module and the other is the neural
compression module used for noise-robust feature learning.

Feature fusion. With the noise-robust features $\tilde{c}_{t}$ and the current
frame $x_{t}$, the final output frame $y_{t}$ is generated through the
restoration module. Besides $y_{t}$, the restoration module part will also
generate the temporal features $c_{t}$ used for next step.

Figure 4: Example of MV comparison. We find MV is more accurate and more
similar to the MV from clean frames after ME refinement. Zoom in for a better
view.

###  4.2 Feature Alignment

To align the temporal features from last step to the current frame, we need to
predict the MV. In our paper, we use the pre-trained optical flow estimation
network SPyNet [35] as our motion estimation module.

However, estimating accurate MV from the degraded frames is quite difficult.
As shown in Fig. 4 (a), the MV without any processing suffers from corruption
and discontinuity, which is not accurate compared with the MV estimated from
clean frames in Fig. 4 (c). To solve this problem, we propose using an MV
refinement module to improve the MV. The MV refinement module adopts a
lightweight auto-encoder structure. It encodes the corrupted MV into compact
representations and then decodes them to the refined MV. The detailed network
structure can be found in appendices. As Fig. 4 (b) shows, with our MV
refinement, the MV is cleaner and more similar to the MV from the clean
frames.

###  4.3 Feature Refinement via Neural Compression

Previous recurrent methods directly fuse the current frame and the aligned
temporal features without any refinement. Actually, the temporal features may
still contain some noisy and uncorrelated information, which disturbs the
restoration of the current frame.

To solve this problem, we propose a feature refinement process to learn noise-
robust feature representations. This process consists of two modules, i.e.
attention module and neural compression module. For the attention mechanism,
many papers [17, 52, 30, 32] have studied it and proved its effectiveness.
Thus, we design an attention module to scale the temporal features to help the
feature learning. To achieve a good trade-off between performance and
complexity, we design an auto-encoder-based attention network whose detailed
network structure can be found in appendices.

After the attention module, the temporal features $\check{c}_{t}$ will be
purified by the proposed neural compression module. Following the design in
neural image/video compression [5, 28, 20], our neural compression module
consists of feature encoder-decoder, quantization process, and a prior model.

First, the temporal features $\check{c}_{t}$ are encoded to be compact latent
codes $e_{t}$ through the feature encoder:

$e_{t}=Encoder(\check{c}_{t}).$ (1)

To achieve the robustness to noise, the quantization is applied to $e_{t}$.
The $e_{t}\in[s_{k},s_{k+1})$ is quantized to value $\frac{s_{k}+s_{k+1}}{2}$,
where $s_{k}$ and $s_{k+1}$ indicate the numerical range. Let
$\ddot{c}_{t}=\check{c}_{t}+\epsilon$ be the noisy feature with noise
$\epsilon$. Under the assumption that $Encoder$ is Lipschitz continuous,
$\ddot{e}_{t}=Encoder(\ddot{c}_{t})$ and $e_{t}=Encoder(\check{c}_{t})$ will
be located in the same region $[s_{k},s_{k+1})$ with high probability if the
quantization step $s_{k+1}-s_{k}$ is relatively large, then they have the same
quantized value. That means, the quantized representation is robust to noisy
input. However, the robustness is determined by a prerequisite that the data
distribution and quantization step are properly learned.

Most existing quantization solutions in neural image/video compression only
use fixed quantization step. Actually, the content characteristics spatially
vary in a large degree. A fixed quantization step cannot handle various and
complex contents well. For example, a fixed small quantization step fails to
remove the noisy information. A fixed large quantization step instead causes
large information loss (i.e., the intrinsic quantization noise). Thus, we
propose an adaptive quantization mechanism, where the quantization step is
learned. The illustration is shown in Fig. 5. First, $e_{t}$ are divided by
the learned quantization step $q_{t}$ after subtracting the learned mean value
$\mu_{t}$. The quotients are then rounded to the closest integers. At last,
the quantized latent codes $\hat{e}_{t}$ are obtained via the opposite
operations. The formulation is:

$\hat{e}_{t}=\lceil\frac{e_{t}-\mu_{t}}{q_{t}}\rfloor*q_{t}+\mu_{t}.$ (2)

$\lceil\cdot\rfloor$ is the integer rounding operation. With the quantized
latent codes $\hat{e}_{t}$, the noise-robust temporal features $\tilde{c}_{t}$
are then decoded through the feature decoder:

$\tilde{c}_{t}=Decoder(\hat{e}_{t}).$ (3)

Figure 5: The illustration of adaptive quantization mechanism. CE means cross-
entropy.

As aforementioned, the data distribution and quantization step need to be
properly learned to achieve noise robustness. In practice, we do not know the
data distribution, thus we use the prior model to estimate it, and then use
the cross-entropy loss to guide the learning of data distribution and
quantization step. The cross-entropy loss is formulated as:

$\mathcal{L}oss_{CE}=E_{\hat{e}_{t}}[-log_{2}p_{\hat{e}_{t}}(\hat{e}_{t})],$
(4)

where $p_{\hat{e}_{t}}(\hat{e}_{t})$ is the estimated probability mass
function of the latent codes $\hat{e}_{t}$. In this paper, we follow [20, 18]
and assume that $p_{\hat{e}_{t}}(\hat{e}_{t})$ follows the Laplace
distribution. The prior model composed by neural network is used for
estimating the distribution parameters. Detailed structures of the prior model
can be found in appendices. But different from [20, 18] that only estimate the
distribution parameters $(\mu_{t},\sigma_{t})$, our prior model also learns
the quantization step size $q_{t}$. With $(\mu_{t},\sigma_{t},q_{t})$, the
probability estimation of $p_{\hat{e}_{t}}(\hat{e}_{t})$ is calculated as:

$p_{\hat{e}_{t}}(\hat{e}_{t})=\prod_{i}(\mathcal{L}(\mu_{t,i},\sigma^{2}_{t,i})*\mathcal{U}(-\frac{q_{t,i}}{2},\frac{q_{t,i}}{2}))(\hat{e}_{t,i}),$
(5)

where $i$ specifies the spatial position of each element in $\hat{e}_{t}$.
According the probability mass function in Eq. 5, we can calculate the cross-
entropy loss via Eq. 4. The cross-entropy loss guides the compression module
to learn proper data distribution and quantization step, and then achieves the
robustness to noise.

In our framework, the quantization step $q_{t}$ is learnable at spatial-
channel-wise. It can be adaptive to regions with different content
characteristics. We visualize one channel example of quantization step map in
Fig. 6. The pixel intensity represents the size of quantization step. The
larger pixel intensity represents the more noisy information that should be
eliminated. As Fig. 6 shows, the quantization step size of the smooth region
is usually larger because the noisy information therein is easier to remove.
By contrast, the quantization step size of texture region (e.g., the table in
the background poster, and there actually exist many details) is usually
smaller.

Figure 6: Visualization example of the input frame and the corresponding
learned quantization step size map. The frame is from Johnny, HEVC Class E
dataset [6].

###  4.4 Feature Fusion

The feature fusion part contains a restoration module. It will fuse the noise-
robust temporal features $\tilde{c}_{t}$ with the current frame $x_{t}$, and
then generates the final output frame $~{}\tilde{y_{t}}$. It is noted that,
besides final output frame $~{}\tilde{y_{t}}$, the restoration module also
generates the temporal features $c_{t}$ used for the next step, like [8]. Our
restoration module is based on a lightweight U-Net [37]. The detailed network
structures can be found in appendices.

$\sigma$ | VNLnet [13] | DVDNet [38] | FastDVDNet [39] | EMVD-L [29] | EMVD-S [29] | EDVR [41] | BasicVSR [8] | BasicVSR++ [9] | Ours |
---|---|---|---|---|---|---|---|---|---|---
10 | 37.10/0.9637 | 36.08/0.9592 | 36.44/0.9624 | 36.56/0.9624 | 35.01/0.9442 | 37.16/0.9658 | 37.12/0.9674 |  37.27//0.9682 |  37.17/0.9684 |
20 | 33.88/0.9360 | 33.49/0.9307 | 33.43/0.9334 | 33.27/0.9320 | 31.65/0.8927 | 34.09/0.9379 | 34.13/0.9397 |  34.25/0.9411 | 34.22/0.9437 |
30 | 31.95/0.9096 | 31.79/0.9023 | 31.68/0.9066 | 31.40/0.9032 | 29.94/0.8678 | 32.31/0.9125 | 32.33/0.9157 |  32.55/0.9168 | 32.57/0.9184 |
40 | 30.55/0.8814 | 30.55/0.8745 | 30.46/0.8812 | 30.05/0.8761 | 28.64/0.8328 | 31.02/0.8887 | 31.05/0.8929 |  31.28/0.8936 | 31.39/0.8970 |
50 | 29.47/0.8561 | 29.56/0.8480 | 29.53/0.8573 | 29.15/0.8528 | 27.83/0.8082 | 30.06/0.8660 | 30.11/0.8690 |  30.32/0.8696 | 30.45/0.8770 |
FLOPs (G) | - | - | 665 | 1106 | 5 | 3089 | 2947 | 3402 | 771 |
|  |  |  |  |  |  |  |  |  |

Table 1: PSNR/SSIM comparison with SOTA video denoising methods on synthetic
dataset Set8. The best performance is highlighted in red (1st best) and blue
(2nd best). Our method achieves the best SSIM on all noise levels.

| FastDVDNet [39] | EDVR [41] | RViDeNet [48] | EMVD-L [29] | EMVD-S [29] | BasicVSR [8] | BasicVSR++ [9] | Ours | Ours-L
---|---|---|---|---|---|---|---|---|---
PSNR | 44.30 | 44.71 | 44.08 | 44.48 | 42.63 | 44.80 | 44.98 | 44.72 | 45.09
SSIM | 0.9881 | 0.9902 | 0.9881 | 0.9895 | 0.9851 | 0.9903 | 0.9903 | 0.9906 | 0.9909
Runtime (ms) | 132 | 1511 | 1254 | 246 | 59 | 425 | 488 | 188 | 275
|  |  |  |  |  |  |  |  |

Table 2: Comparison with SOTA video denoising methods on real-world dataset
CRVD [48]. Our method with default setting outperforms other fast methods and
gets close to the slow method. With a more powerful restoration module (i.e.
’Ours-L’), we can achieve SOTA performance in terms of both PSNR and SSIM. The
runtime is the average frame runtime for the whole dataset on a P100 GPU.

|  | MS-CSC [21] | SE [42] | SpacCNN [12] | FastDerain [19] | J4RNet-P [25] | FCRVD [46] | RMFD [47] | BasicVSR++ [9] | Ours
---|---|---|---|---|---|---|---|---|---|---
RainSynAll100 | PSNR | 16.19 | 15.29 | 18.39 | 17.09 | 19.26 | 21.06 | 25.14 | 27.67 | 28.11
SSIM | 0.5078 | 0.5053 | 0.6469 | 0.5824 | 0.6238 | 0.7405 | 0.9172 | 0.9135 | 0.9235
RainSynComplex25 | PSNR | 16.96 | 16.76 | 21.21 | 19.25 | 24.13 | 27.72 | 32.70 | 33.42 | 34.27
SSIM | 0.5049 | 0.5273 | 0.5854 | 0.5385 | 0.7163 | 0.8239 | 0.9357 | 0.9365 | 0.9434
|  |  |  |  |  |  |  |  |  |

Table 3: Comparison with SOTA video deraining methods on RainSynComplex25 [25]
and RainSynAll100 [47]. We train the BasicVSR++ [9] using the same setting as
ours. Other baseline results are provided by RMFD [47] paper.

| DCP [15] | GDNet [26] | DuRN [27] | KDDN [16] | MSBDN [14] | FFA [34] | VDN [36] | EDVR [41] | CG-IDN [50] | BasicVSR++ [9] | Ours
---|---|---|---|---|---|---|---|---|---|---|---
PSNR | 11.03 | 19.69 | 18.51 | 16.32 | 22.01 | 16.65 | 16.64 | 21.22 | 23.21 | 21.68 | 23.63
SSIM | 0.7285 | 0.8545 | 0.8272 | 0.7731 | 0.8759 | 0.8133 | 0.8133 | 0.8707 | 0.8836 | 0.8726 | 0.8925
|  |  |  |  |  |  |  |  |  |  |

Table 4: Comparison with SOTA video dehazing methods on REVIDE [50] testset.
We train the BasicVSR++ [9] using the same setting as ours. Other baseline
results are provided by CG-IDN [50] paper.

###  4.5 Loss Function

In our method, the loss function includes two items:

$\mathcal{L}oss=\sum_{t=1}^{n}\mathcal{L}oss_{\mathcal{L}2}(y_{t},~{}\tilde{y_{t}})+\lambda\cdot\mathcal{L}oss_{CE}(\hat{e}_{t}).$
(6)

$y_{t}$ and $~{}\tilde{y_{t}}$ are the clean and estimated frames,
respectively. $\mathcal{L}oss_{\mathcal{L}2}$ is the L2 loss and
$\mathcal{L}oss_{CE}$ is the cross-entropy loss. To learn noise-robust feature
representation and then let it help the final reconstruction, we adopt a two-
stage training scheme and the details are in appendices.

##  5 Experiment

We evaluate our method on several video restoration tasks, including
denoising, deraining, and dehazing.

###  5.1 Dataset

Video denoising. Both synthetic dataset and real-world dataset are tested. For
the synthetic dataset, we follow the setting in FastDVDNet [39]. DAVIS2017
train-val set containing 90 videos is used for training. Set8 is used for
testing. We add the additive white Gaussian noise (AWGN) to the clean video to
synthesize the noisy video. Five noise levels, i.e. $\sigma$=10, 20, 30, 40,
50 are tested. For the real-world dataset, we follow the setting in EMVD [29]
and use the dataset from RViDeNet [48]. It consists of a captured raw video
dataset (CRVD) and a synthetic raw video dataset (SRVD). Following EMVD and
RviDeNet, we use CRVD scene 1$\sim$6 plus SRVD for training and CRVD scene
7$\sim$11 for testing.

Video deraining. Following [47], we test our method on RainSynComplex25 [25]
and RainSynAll100 [47] datasets. RainSynComplex25 contains 190 videos for
training and 25 videos for testing. RainSynAll100 contains 900 videos for
training and 100 videos for testing.

Video dehazing. We use REVIDE [50] dataset which captures the pairs of hazy
and corresponding haze-free videos in the same scene by an acquisition system.
It contains 42 videos for training and 6 videos for testing.

###  5.2 Result on Video Denoising

We compare our method with these baselines: VNLNet [13], DVDNet [38],
FastDVDNet [39], EMVD [29], EDVR [41], BasicVSR [8], BasicVSR++ [9], and
RViDeNet [48]. EMVD has several network structure configurations with
different complexities. The large (EDVR-L) and small (EMVD-S) models are
tested (more details about configurations are in appendices). The original
BasicVSR/BasicVSR++ are bi-directional methods that leverage the temporal
features from both the future and the past frames. To compare with other
methods more fairly, we modified BasicVSR/BasicVSR++ to uni-directional
methods that only use the temporal features from the past frames.

Quantitative Comparison. We use peak signal-to-noise ratio (PSNR) and
structural similarity index measure (SSIM) as quantitative evaluation metrics.
We present the results on synthetic noise video in Table 1 and the results on
real-world noisy video in Table 2. For the synthetic video, as Table 1 shows,
our method achieves the best SSIM on all noise levels. For PSNR, our method
outperforms the second-best method BasicVSR++ [9] with as least 0.11dB gain
when the noise level is high ($\sigma$ = 40 or 50). In addition, we find the
quality improvement over BasicVSR++ is larger when the noise level is higher.
It verifies that our proposed neural compression module can effectively filter
the noise. It is also worth noting that the FLOPs of our method is only 0.23
times of BasicVSR++, which shows that our method achieves a much better trade-
off between quality and complexity. Compared with the low-complexity methods
FastDVDNet [39] and EMVD-L [29], our method achieves significant quality
improvement. For the real-world noisy video, it should be admitted that our
method with default setting currently cannot outperform BasicVSR and
BasicVSR++ in terms of PSNR but is better than them in terms of SSIM. When
compared with low-complexity method FastDVDNet and EMVD-L, our method can
achieve the best quality. Besides, if we change our U-Net-like [37]
restoration network to a W-Net-like [43] restoration network with more
complexity (more details are in appendices), denoted as ’Our-L’ in Table 2, we
can achieve the best PSNR and SSIM at the same time, but the complexity is
still much less than that of BasicVSR and BasicVSR++.

Qualitative Comparison. Fig. 7 shows the visual quality comparison. As Fig. 7
shows, FastDVDNet without feature alignment suffers from serious distortion in
the text region. The results of BasicVSR++ are quite blurry caused by the
propagated noise in temporal features. By contrast, our neural compression-
based method can learn noise-robust features and is able to restore much
clearer textures. More visual comparisons are in appendices.

Figure 7: Denoising: motorbike video from Set8 testset with noise variance 50.
Deraining: 0985 video from RainSynAll100 testset. Dehazing: L006 video from
REVIDE dataset.

| $M_{a}$ | $M_{b}$ | $M_{c}$ | $M_{d}$
---|---|---|---|---
MVR |  | ✓ | ✓ | ✓
NCFL |  |  | ✓ | ✓
FA |  |  |  | ✓
PSNR | 29.75 | 29.87 | 30.29 | 30.45
|  |  |  |

Table 5: The ablation study on different modules. Tested on Set8
($\sigma=50$). MVR is the MV refinement, NCFL is the neural compression-based
feature learning. FA is the feature attention.

| NCFL-AdapQ | NCFL-FixedQ | NCFL-NoQ
---|---|---|---
PSNR | 30.29 | 29.86 | 29.98
|  |  |

Table 6: The ablation study on quantization. Tested on Set8 ($\sigma=50$).
NCFL-AdapQ is our default model with adaptive quantization, i.e. $M_{c}$ in
Table 5. NCFL-FixedQ means we use the fixed quantization step as in existing
neural video codecs. NCFL-NoQ removes the quantization and is only a vanilla
auto-encoder.

###  5.3 Result on Video Deraining

We compare our method with prior SOTA video deraining methods, including MS-
CSC [21], SE [42], SpacCNN [12], FastDerain [19], J4RNet-P [25], FCRVD [46],
RMFD [47], and BasicVSR++ [9]. Since RainSynAll100 uses the rain accumulation
degradation to generate the rainy video, parts of the baseline methods
including SE, MS-CSC, SpacCNN, and FastDerain could not handle this
degradation thus MRF [7] is used as post-processing. More details could be
found in [47]. FCRVD, RMFD, BasicVSR++, and our method could handle this
degradation without extra post-processing. As Table 3 shows, BasicVSR++ beats
RMFD in terms of PSNR and SSIM on RainSynComplex25 but is slightly worse than
RMFD in terms of SSIM on RainSynAll100. By contrast, with the robust temporal
features, our method achieves the best PSNR and SSIM on both datasets. Our
method brings PSNR gain of 0.44 dB and SSIM gain of 0.0063 on RainSynAll100.
We also test RainSynLight25 [25] and NTURain [12]. Their results are provided
in appendices. Fig. 7 also shows the visual quality comparison. We can see
that our model could remove rain streak well, and produces clearer and more
visual pleasing results.

###  5.4 Result on Real-World Video Dehazing

Table 4 shows the comparison between our method and prior SOTA real-world
video dehazing method: DCP [15], GDNet [26], DuRN [27], KDDN [16], MSBDN [14],
FFA [34], VDN [36], EDVR [41], CG-IDN [50], and BasicVSR++ [9]. As Table 4
shows, BasicVSR++ outperforms EDVR but is worse than MBSDN and CG-IDN that are
specially designed for dehazing task. By contrast, our method has 0.42 dB PSNR
and 0.0089 SSIM improvements when compared with the second-best method CG-IDN.
In addition, the parameters of our method are 16M, and only 0.70 times of CG-
IDN that has 23M parameters. As Fig. 7 shows, the result of our method is more
visual pleasing.

###  5.5 Ablation Study

This paper proposes three key modules: the MV refinement (MVR) for improving
MV, the neural compression-based feature learning (NCFL) incorporated with
adaptive quantization, and feature attention (FA). We study the effect of
these modules and report the results in Table 5. Without MVR, NCFL, and FA,
the baseline model only contains a motion estimation module, bilinear warping,
and a restoration module.

MV refinement (MVR). As Table 5 shows, the baseline model $M_{a}$ only
achieves PSNR 29.75dB. It suffers from the discontinuous MV estimated from
noisy video. When enabling our MVR, the MV is refined and $M_{b}$ reaches PSNR
29.87 dB. Our MVR brings 0.12 dB PSNR improvement.

Neural compression-based feature learning (NCFL). If we further combine NCFL
and MVR, $M_{c}$ reaches PSNR 30.29 dB and improves 0.42 dB compared with
$M_{b}$. The significant improvement verifies that the effectiveness of NCFL.
In addition, we also study two variants of NCFL. As Table 6 shows, the PSNR of
a vanilla auto-encoder without quantization (i.e. NCFL-NoQ) drops to 29.98dB.
This denotes that the improvement brought by NCFL mainly comes from the
adaptive quantization mechanism rather than the increase of model parameters.
In addition, we also test NCFL-FixedQ where a fixed quantization step is used
as many existing neural video codecs do. The PSNR of NCFL-FixedQ drops to
29.89 dB. Its performance is even worse than NCFL-NoQ. This shows that a fixed
quantization step instead loses some useful information and fails at learning
noise-robust representations. By contrast, a learnable quantization step at
spatial-channel-wise can adaptively filter the noise and purify the temporal
features with different content characteristics, which is quite important.

Feature attention (FA). In this paper, we also propose a FA module to further
help the feature learning. As Table 5 shows, $M_{d}$ achieves PSNR 30.45 dB.
FA boosts the PNSR by 0.16 dB, which shows its effectiveness.

###  5.6 NCFL on Different Degradations

Table 5 and Table 6 invesitage NCFL under AWGN degradation. However, our NCFL
is not confined to AWGN. It is also very effective for other complex
degradations, such as real-word denoising, deraining, and dehazing. Table 7
shows the comprehensive study on multiple degradations. For example, the
comparison between $M_{1}$ and $M_{3}$ shows that NCFL can achieve 0.81 dB
gain for deraining. These substantial improvements verify the effectiveness of
our NCFL. In addition, the comparison between $M_{2}$ and $M_{3}$ shows that
cross-entropy loss can effectively guides the learning of NCFL under multiple
degradations.

| Deraining | Dehazing | RWD | AWGN
---|---|---|---|---
$M_{1}$: w/o NCFL | 27.30 | 23.07 | 44.48 | 29.99
$M_{2}$: w/ NCFL (w/o CE) | 27.64 | 23.30 | 44.56 | 30.20
$M_{3}$: w/ NCFL | 28.11 | 23.63 | 44.72 | 30.45
|  |  |  |

Table 7: Study on NCFL under different degradation types. CE means the cross-
entropy loss. RWD means real-world denoising. AWGN indicates additive white
Gaussian noise.

###  5.7 Bi-directional Video Denoising

In previous experiments, we focus on the uni-directional setting where the
temporal features only come from the past time. For the bi-directional
setting, both the temporal features come from the past time and the future
time can be used. One advantage of our method is that our method can be
extended to bi-directional setting easily. We test the bi-directional model of
BasicVSR [8], BasicVSR++ [9], and our bi-directional model. The PSNR and
complexity comparison is shown in Table 8. As Table 8 shows, the bi-
directional setting brings BasicVSR 0.59 dB gain, BasicVSR++ 0.78 dB gain, and
our method 0.76 dB gain with about 2x complexity. Under bi-directional
setting, our method still outperforms BasicVSR++ by 0.11 dB with only 0.21x
FLOPs.

Direction | Method | PSNR | FLOPs (G)
---|---|---|---
Uni-direction | BasicVSR | 30.11 | 2947
Uni-direction | BasicVSR++ | 30.32 | 3402
Uni-direction | Ours | 30.45 | 771
Bi-direction | BasicVSR | 30.68 | 5855
Bi-direction | BasicVSR++ | 31.10 | 7097
Bi-direction | Ours | 31.21 | 1522
|  |  |

Table 8: Bidirectional video denoising on Set8 with $\sigma$ = 50.

##  6 Conclusion and Limitation

In this paper, we have designed a neural compression-based video restoration
framework. We are inspired by the fact that neural video codec can naturally
filter the noise, and then propose using neural compression to purify the
temporal features and learn noise-robust feature representations. To solve the
problem that the fixed quantization step harms the inherent textures, we
propose a learnable quantization mechanism at spatial-channel-wise to achieve
robustness to noise. At the same time, an attention module and an MV
refinement module are proposed to further boost the performance. Experimental
results show that the proposed method achieves a much better quality-
complexity trade-off than previous SOTA methods.

Although our method is faster than most previous SOTA methods, the inference
speed of our method still does not meet the requirements of real-time
scenarios. In the future, we will continue to improve the efficiency of our
method for real-time video restoration.

## References

  * [1] Ffmpeg.  https://www.ffmpeg.org/.
  * [2] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici.  Scale-space flow for end-to-end optimized video compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8503–8512, 2020.
  * [3] Johannes Ballé, Valero Laparra, and Eero P Simoncelli.  Density modeling of images using a generalized normalization transformation.  arXiv preprint arXiv:1511.06281, 2015.
  * [4] Johannes Ballé, Valero Laparra, and Eero P Simoncelli.  End-to-end optimized image compression.  arXiv preprint arXiv:1611.01704, 2016.
  * [5] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston.  Variational image compression with a scale hyperprior.  arXiv preprint arXiv:1802.01436, 2018.
  * [6] Frank Bossen et al.  Common test conditions and software reference configurations.  In JCTVC-L1100, 2013.
  * [7] Bolun Cai, Xiangmin Xu, and Dacheng Tao.  Real-time video dehazing based on spatio-temporal mrf.  In Pacific Rim conference on multimedia, pages 315–325. Springer, 2016.
  * [8] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  BasicVSR: The search for essential components in video super-resolution and beyond.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4947–4956, 2021.
  * [9] Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy.  BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment.  arXiv preprint arXiv:2104.13371, 2021.
  * [10] Baymax Chen.  EMVD, https://github.com/baymax-chen/emvd, 2021.
  * [11] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.  Learning to see in the dark.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3291–3300, 2018.
  * [12] Jie Chen, Cheen-Hau Tan, Junhui Hou, Lap-Pui Chau, and He Li.  Robust video content alignment and compensation for rain removal in a cnn framework.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6286–6295, 2018.
  * [13] Axel Davy, Thibaud Ehret, Jean-Michel Morel, Pablo Arias, and Gabriele Facciolo.  Non-local video denoising by cnn.  arXiv preprint arXiv:1811.12758, 2018.
  * [14] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang.  Multi-scale boosted dehazing network with dense feature fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2157–2167, 2020.
  * [15] Kaiming He, Jian Sun, and Xiaoou Tang.  Single image haze removal using dark channel prior.  IEEE transactions on pattern analysis and machine intelligence, 33(12):2341–2353, 2010.
  * [16] Ming Hong, Yuan Xie, Cuihua Li, and Yanyun Qu.  Distilling image dehazing with heterogeneous task imitation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3462–3471, 2020.
  * [17] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [18] Zhihao Hu.  Pytorch video compression, https://github.com/zhihaohu/pytorchvideocompression, 2020.
  * [19] Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, and Yao Wang.  Fastderain: A novel video rain streak removal method using directional gradient priors.  IEEE Transactions on Image Processing, 28(4):2089–2102, 2018.
  * [20] Jiahao Li, Bin Li, and Yan Lu.  Deep contextual video compression.  Advances in Neural Information Processing Systems, 34, 2021.
  * [21] Minghan Li, Qi Xie, Qian Zhao, Wei Wei, Shuhang Gu, Jing Tao, and Deyu Meng.  Video rain streak removal by multiscale convolutional sparse coding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6644–6653, 2018.
  * [22] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [23] Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu.  M-LVC: multiple frames prediction for learned video compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [24] Jerry Liu, Shenlong Wang, Wei-Chiu Ma, Meet Shah, Rui Hu, Pranaab Dhawan, and Raquel Urtasun.  Conditional entropy coding for efficient video compression.  arXiv preprint arXiv:2008.09180, 2020.
  * [25] Jiaying Liu, Wenhan Yang, Shuai Yang, and Zongming Guo.  Erase or fill? deep joint recurrent rain removal and reconstruction in videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3233–3242, 2018.
  * [26] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen.  Griddehazenet: Attention-based multi-scale network for image dehazing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7314–7323, 2019.
  * [27] Xing Liu, Masanori Suganuma, Zhun Sun, and Takayuki Okatani.  Dual residual networks leveraging the potential of paired operations for image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7007–7016, 2019.
  * [28] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao.  DVC: an end-to-end deep video compression framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11006–11015, 2019.
  * [29] Matteo Maggioni, Yibin Huang, Cheng Li, Shuai Xiao, Zhongqian Fu, and Fenglong Song.  Efficient multi-stage video denoising with recurrent spatio-temporal fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3466–3475, June 2021.
  * [30] Yiqun Mei, Yuchen Fan, Yulun Zhang, Jiahui Yu, Yuqian Zhou, Ding Liu, Yun Fu, Thomas S Huang, and Humphrey Shi.  Pyramid attention networks for image restoration.  arXiv preprint arXiv:2004.13824, 2020.
  * [31] David Minnen, Johannes Ballé, and George Toderici.  Joint autoregressive and hierarchical priors for learned image compression.  arXiv preprint arXiv:1809.02736, 2018.
  * [32] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen.  Single image super-resolution via a holistic attention network.  In European Conference on Computer Vision, pages 191–207. Springer, 2020.
  * [33] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In Computer Vision and Pattern Recognition, 2016.
  * [34] Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia.  FFA-Net: Feature fusion attention network for single image dehazing.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11908–11915, 2020.
  * [35] Anurag Ranjan and Michael J Black.  Optical flow estimation using a spatial pyramid network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4161–4170, 2017.
  * [36] Wenqi Ren, Jingang Zhang, Xiangyu Xu, Lin Ma, Xiaochun Cao, Gaofeng Meng, and Wei Liu.  Deep video dehazing with semantic segmentation.  IEEE Transactions on Image Processing, 28(4):1895–1908, 2018.
  * [37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [38] Matias Tassano, Julie Delon, and Thomas Veit.  DVDnet: A fast network for deep video denoising.  In 2019 IEEE International Conference on Image Processing (ICIP), pages 1805–1809. IEEE, 2019.
  * [39] Matias Tassano, Julie Delon, and Thomas Veit.  Fastdvdnet: Towards real-time deep video denoising without flow estimation.  In CVPR, pages 1354–1363, 2020.
  * [40] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of machine learning research, 9(11), 2008.
  * [41] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy.  EDVR: Video restoration with enhanced deformable convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [42] Wei Wei, Lixuan Yi, Qi Xie, Qian Zhao, Deyu Meng, and Zongben Xu.  Should we encode rain streaks in video as deterministic or stochastic?  In Proceedings of the IEEE International Conference on Computer Vision, pages 2516–2525, 2017.
  * [43] Xide Xia and Brian Kulis.  W-net: A deep model for fully unsupervised image segmentation.  arXiv preprint arXiv:1711.08506, 2017.
  * [44] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision, 127(8):1106–1125, 2019\.
  * [45] Bo Yan, Chuming Lin, and Weimin Tan.  Frame and feature-context video super-resolution.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5597–5604, 2019.
  * [46] Wenhan Yang, Jiaying Liu, and Jiashi Feng.  Frame-consistent recurrent video deraining with dual-level flow.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1661–1670, 2019.
  * [47] Wenhan Yang, Robby T Tan, Jiashi Feng, Shiqi Wang, Bin Cheng, and Jiaying Liu.  Recurrent multi-frame deraining: Combining physics guidance and adversarial learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [48] Huanjing Yue, Cong Cao, Lei Liao, Ronghe Chu, and Jingyu Yang.  Supervised raw video denoising with a benchmark dataset on dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2301–2310, 2020.
  * [49] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Multi-stage progressive image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14821–14831, 2021.
  * [50] Xinyi Zhang, Hang Dong, Jinshan Pan, Chao Zhu, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Fei Wang.  Learning to restore hazy video: A new real-world dataset and a new method.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9239–9248, 2021.
  * [51] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Proceedings of the European conference on computer vision (ECCV), pages 286–301, 2018.
  * [52] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu.  Residual non-local attention networks for image restoration.  In International Conference on Learning Representations, 2019.

Appendices includes detailed network architecture, training details for
different tasks, and additional ablation studies. More comparisons on video
denoising, video deraining and video dehazing are also presented.

Figure 8: The structure of MV refinement module and feature attention module.
Figure 9: The structure of neural compression-based feature learning module.
Figure 10: The structure of U-Net-like restoration module and CAB block.
Figure 11: The structure of W-Net-like restoration module.

##  Appendix A Network Architecture

Our framework contains three parts: feature alignment, feature refinement
(including feature attention module and neural compression-based feature
learning module), and feature fusion. In this appendix, we present the network
architecture details.

MV refinement module. The structure of the MV refinement module is shown in
Fig. 8 (a). The MV refinement module encodes the corrupted MV into a compact
representation and then decodes it to the refined MV. Specifically, we use two
convolutional layers with stride=2 to encode the corrupted MV and two deconv
layers to decode the refined MV. The number of the channel of the intermediate
features is 64. As GDN [3] could reduce the statistical dependencies of the
features and compact the features, we use GDN [3] in the encoder and use the
inverse-GDN (IGDN) in the decoder correspondingly.

Feature attention module. As Fig. 8 (b) shows, the feature attention module
takes the noisy frame $x_{t}$ and aligned features $\hat{c}_{t}$ as input,
then generates the spatial-channel attention map $m_{t}$. The feature
attention module is based on an auto-encoder with the following modifications.
First, we use Resblock [22] to extract the features at each scale. Second, we
add 4 Resblocks at the largest scale to increase the receptive field with
minor computation cost. To keep the feature attention module lightweight, the
numbers of the channels of the intermediate features are 16, 32, 64 for three
scales, respectively.

Neural compression-based feature learning module. As Fig. 9 shows, the neural
compression-based feature learning module contains a prior model, a feature
encoder, and a feature decoder. The prior model learns to estimate the
parameters $(\mu_{t},\sigma_{t},q_{t})$ that are used in the adaptive
quantization and the $\mathcal{L}oss_{CE}$. The feature encoder encodes the
features $\check{c}_{t}$ into compact latent codes $e_{t}$. Then the latent
codes $e_{t}$ are processed by our proposed adaptive quantization. At last,
the refined features $\tilde{c}_{t}$ are decoded from the processed latent
codes $\hat{e}_{t}$. For the prior model, we use three convolutional layers
(two layers are with stride=2) to estimate the parameters, where LeakyReLU is
used as the activation function. For the feature encoder and decoder, we use
two convolutional layers with stride=2 to encode the features and two
deconvolutional layers to decode the noise-robust features. The number of the
channel of the intermediate features is 64.

Restoration module. As Fig. 10 shows, the restoration module fuses the noise-
robust features $\tilde{c}_{t}$ with the current frame $x_{t}$, and then
generates the final output frame $~{}\tilde{y}_{t}$. At the same time, the
features $c_{t}$ used by next step are also generated. Following previous
image restoration methods [14, 11], we adopt a deep U-Net [37] as our
restoration module for synthetic video denoising, video deraining and video
dehazing. We also leverage the channel-attention block [51] (CAB) to extract
the features at each scale. The structure of CAB is shown in Fig. 10 (b) and
the GAP means the global average pooling. The features from the skip
connection are processed by a convolutional layer. The numbers of the channels
of the intermediate features are 32, 64, 128, 256, 512 for five scales,
respectively. We also propose a more powerful W-Net-like [43] restoration
module for real-world video denoising which is a more challenging task. As
Fig. 11 shows, we cascade two U-Net restoration modules as our W-Net-like
restoration module. The number of the channel of the intermediate features of
each scale remains unchanged.

##  Appendix B Training Details

We adopt AdamW optimizer and cosine annealing learning rate scheduler. The
initial learning rates of the motion estimation module and other modules are
set to 2.5e-5 and 2e-4, respectively. The weights of the motion estimation
module are fixed during the first 2500 iterations. The batch size is 16 and
each sample in the batch is a 5-frame video clip. The patch size is 128x128.
The data augmentation includes random horizontal, vertical, and transposed
flipping.

During the training, we use a two-stage training scheme to learn the noise-
robust feature representation and then make this feature representation help
the final reconstruction. In the first stage, we use the
$\mathcal{L}oss_{\mathcal{L}2}$ and $\mathcal{L}oss_{CE}$ to train the model
for 50K iterations. The first stage training not only provides the feature
generation a good starting point but also helps the compression module
converge to a relatively stable status which can effectively filter the noisy
and irrelevant information. The second stage only use
$\mathcal{L}oss_{\mathcal{L}2}$ for the rest iterations. With a well-trained
compression module, the second stage carefully fine-tunes the quantization
step size and the mean value only guided by the
$\mathcal{L}oss_{\mathcal{L}2}$. This helps the model focus more on the
model’s generation ability for better reconstruction quality. This simple two-
stage training helps the temporal features be robust to the noise, and then
lets these features improve the final quality.

The main difference of training setting among different datasets is the number
of total iterations. For DAVIS [33] dataset and CRVD [48] dataset, we train
our model for 100K iterations and 200K iterations, respectively. For
RainSynLight25 [25], RainSynComplex25 [25], RainSynAll100 [47] and NTU-Rain
[12], we train our model for 200K, 200K, 250K and 100K iterations,
respectively. For REVIDE [50], we train our model for 50K iterations. Besides,
we follow the setting in CG-IDN [50] and use patch size 384x384 in our method
for REVIDE.

##  Appendix C More Ablation Studies

We conduct more ablation studies about the effect of different modules, the
weight of the cross-entropy loss, and the effect of the neural compression-
based feature learning on clean features.

###  C.1 The Effect of Different Modules

This paper proposes three key modules: the MV refinement (MVR) , the neural
compression-based feature learning (NCFL) incorporated with adaptive
quantization, and the feature attention (FA). We study the effect of these
modules and report the results in the main paper. In this appendix, we conduct
the experiments by using normal convolutional layers (without auto-encoder
structure) to replace MVR and NCFL, for demonstrating that the improvements of
our modules are not from increasing complexity or deeper structure. MVR is
replaced by normal convolutional layers, denoted as M-Conv. NCFL is replaced
by normal convolutional layers, denoted as N-Conv. The complexity of M-Conv
and N-Conv is slightly higher than that of MVR and NCFL, respectively. As
Table 9 shows, replacing MVR with M-Conv causes a 0.08 dB PSNR drop
($M_{b}\rightarrow M_{c}$) and replacing NCFL with N-Conv causes a 0.36 dB
PSNR drop ($M_{d}\rightarrow M_{e}$). These results verify that the
improvement of performance is not brought by the increasing complexity but
comes from our special design.

| $M_{a}$ | $M_{b}$ | $M_{c}$ | $M_{d}$ | $M_{e}$ | $M_{f}$
---|---|---|---|---|---|---
MVR |  | ✓ |  | ✓ | ✓ | ✓
M-Conv |  |  | ✓ |  |  |
NCFL |  |  |  | ✓ |  | ✓
N-Conv |  |  |  |  | ✓ |
FA |  |  |  |  |  | ✓
PSNR | 29.75 | 29.87 | 29.79 | 30.29 | 29.93 | 30.45
GFLOPs | 534 | 548 | 557 | 613 | 639 | 771
|  |  |  |  |  |

Table 9: The ablation study on different modules. Tested on Set8
($\sigma=50$). MVR denotes the MV refinement. NCFL denotes the neural
compression-based feature learning. FA denotes the feature attention. M-Conv
represents that we use the normal convolutional layers with slightly larger
complexity than MVR to replace MVR. N-Conv represents that we use the normal
convolutional layers with slightly larger complexity than NCFL to replace
NCFL.

###  C.2 The Weight of Cross-Entropy Loss

In the first stage training, the weight $\lambda$ of the cross-entropy loss
$\mathcal{L}oss_{CE}$ controls the intensity of filtering the noisy
information in the temporal features. As Table 10 shows, if $\lambda$ is large
(e.g., $\lambda$ = 1/256 or 1/512), some useful information may also be
filtered, which degrades performance a bit. On the contrary, if $\lambda$ is
too small ( e.g., $\lambda$ = 1/4096), the temporal features may still contain
some noisy information and performance also drops. The optimal $\lambda$ may
be related to the noise level, i.e. different noise levels require different
$\lambda$ to achieve the best performance. To avoid too many parameters
tuning, we use 1/2048 as the default value of $\lambda$ for other experiments.
In the future, we will investigate how to adaptively set $\lambda$ according
to the noise level and content characteristic.

$\lambda$ | 1/256 | 1/512 | 1/1024 | 1/2048 | 1/4096
---|---|---|---|---|---
PSNR | 30.25 | 30.29 | 30.35 | 30.45 | 30.40
|  |  |  |  |
Table 10: The ablation study on the weight $\lambda$ of
$\mathcal{L}oss_{Entropy}$. Tested on Set8 ($\sigma=50$).

###  C.3 The Effect of Neural Compression-Based Feature Learning on Clean
Features

Our neural compression-based feature learning is used to filter the noisy and
irrelevant information in noisy features, and it will not discard useful
information in the clean features. To verify it, we use clean frames from
DAVIS train-val dataset as input and groundtruth to train the individual NCFL
module with adaptive quantization (NCFL-AdapQ) and the NCFL module without
quantization (NCFL-NoQ). We do not train our complete model because we
directly use the current input frame as the input of restoration module. This
will enable the restoration module to directly learn the identity mapping when
the input is clean, and make the temporal feature propagation useless. The
results show that NCFL-AdapQ reaches PSNR 30.37 dB and NCFL-NoQ reaches PSNR
30.39 dB. The similar results of these two models show that the proposed
adaptive quantization has little impact on the clean features.

$\sigma$ | VNLnet [13] | DVDNet [38] | FastDVDNet [39] | EMVD-L [29] | EMVD-S [29] | EDVR [41] | BasicVSR [8] | BasicVSR++ [9] | Ours |
---|---|---|---|---|---|---|---|---|---|---
10 | 35.83/0.9473 | 38.13/0.9679 | 38.71/0.9702 | 38.57/0.9695 | 36.90/0.9512 | 39.23/0.9732 | 39.55/0.9758 |  39.71//0.9761 |  39.67/0.9782 |
20 | 34.49/0.9231 | 35.70/0.9470 | 35.77/0.9468 | 35.39/0.9413 | 33.58/0.9023 | 36.33/0.9516 | 36.65/0.9558 |  36.75/0.9565 | 36.78/0.9596 |
30 | 33.42/0.9086 | 34.08/0.9255 | 34.04/0.9252 | 33.89/0.9210 | 31.94/0.8878 | 34.62/0.9311 | 35.07/0.9389 |  35.21/0.9403 | 35.24/0.9435 |
40 | 32.32/0.8974 | 32.86/0.9040 | 32.82/0.9047 | 32.40/0.8941 | 30.66/0.8508 | 33.40/0.9113 | 33.73/0.9207 |  33.96/0.9229 | 34.06/0.9267 |
50 | 31.43/0.8761 | 31.85/0.8829 | 31.86/0.8851 | 31.47/0.8747 | 29.88/0.8273 | 32.41/0.8937 | 32.81/0.9046 |  32.93/0.9056 | 33.11/0.9107 |
GFLOPs | - | - | 665 | 1106 | 5 | 3089 | 2947 | 3402 | 771 |
|  |  |  |  |  |  |  |  |  |

Table 11: PSNR/SSIM comparison with SOTA video denoising methods on DAVIS
testset. The best performance is highlighted in red (1st best) and blue (2nd
best). Our method achieves the best SSIM on all noise levels. Figure 12: The
denoised results of hypersmooth from Set8 testset with noise variance 50. Best
viewed in color.  Figure 13: The denoised results of park_joy from Set8 test
set with noise variance 50. Best viewed in color.

|  | MS-CSC [21] | SE [42] | SpacCNN [12] | FastDerain [19] | J4RNet-P [25] | FCRVD [46] | RMFD [47] | BasicVSR++ [9] | Ours
---|---|---|---|---|---|---|---|---|---|---
NTU-Rain | PSNR | 27.31 | 25.73 | 33.11 | 30.32 | 32.14 | 36.05 | 38.92 | 39.48 | 40.22
SSIM | 0.7870 | 0.7614 | 0.9474 | 0.0.9262 | 0.9480 | 0.9676 | 0.9764 | 0.9776 | 0.9811
RainSynLight25 | PSNR | 25.58 | 26.56 | 32.78 | 29.42 | 32.96 | 35.80 | 36.99 | 38.56 | 39.17
SSIM | 0.8089 | 0.8006 | 0.9239 | 0.8683 | 0.9434 | 0.9622 | 0.9760 | 0.9813 | 0.9872
|  |  |  |  |  |  |  |  |  |

Table 12: Comparison with SOTA video deraining methods on NTU-Rain [12] and
RainSynLight25 [25]. The best performance is highlighted in red (1st best) and
blue (2nd best). We train the BasicVSR++ [9] using the same setting as ours.
Other baseline results are provided by RMFD [47] paper. Figure 14: The results
of 0995 video from RainSynAll100 testset . Best viewed in color.  Figure 15:
The results of W002 video from REVIDE testset. Best viewed in color.

##  Appendix D More Comparisons on Video Denosing

We compare our method with these baselines: VNLNet [13], FastDVDNet [39], EMVD
[29], EDVR [41], BasicVSR [8], and BasicVSR++ [9]. EMVD has several network
structure configurations with different complexities. More specifically, EMVD
mainly contains three modules: the fusion module, the denoising module, and
the refinement module. The number of convolutional layers and the number of
the channel of the intermediate features determine the complexity. The default
setting of EMVD is (f-2-16, d-2-16, r-2-16). ’f-2-16’ means that the fusion
module contains 2 convolutional layers and the number of the channel of the
intermediate feature is 16. ’d-2-16’ represents the configuration for the
denoising module and ’r-2-16’ represents the configuration for the refinement
module. We denote the default setting as EMVD-small (EMVD-S, f-2-16, d-2-16,
r-2-16). In addition, we also compare another setting, i.e. EMVD-large
(EDVR-L, f-4-64, d-6-256, r-4-64). The two settings can be found in the Table
1(b) in EMVD paper [29]. Since the official code of EMVD is not released, we
use the third-party implementation [10].

|  $\sigma$=10 |  $\sigma$=20 |  $\sigma$=30 |  $\sigma$=40 |  $\sigma$=50
---|---|---|---|---|---
Median filtering (3x3) | 27.87 | 22.45 | 19.91 | 18.34 | 15.95
Median filtering (5x5) | 26.63 | 21.80 | 19.43 | 17.92 | 15.65
Ours | 37.12 | 34.22 | 32.57 | 31.39 | 30.45
|  |  |  |  |

Table 13: PSNR comparison with median filtering under AWGN, Set8 testset.

Quantitative Comparison. In the main paper, we have shown the results on Set8
testset when compared with previous neural network-based methods. In addition,
to help us better understand the advantage of neural network-based method over
traditional method, we also test median filtering and summarize the comparison
in Table 13. From this table, we can see that our method achieves significant
quality improvement over median filtering under all noise levels.

In this appendix, we also provide the PSNR/SSIM results for DAVIS testset [33]
in Table 11. As Table 11 shows, our method outperforms other methods on high
noise levels ($\sigma$ = 20, 30, 40, and 50) in terms of PSNR and outperforms
them on all noise levels in terms of SSIM. With the increasing of noise level
, the difference of performance between our method and BasicVSR++ also becomes
larger. When $\sigma$ is 10, 20, 30, 40, and 50, the difference of PSNR
between our method and BasicVSR++ is -0.04, 0.03, 0.03, 0.10 and 0.18,
respectively. These results demonstrate that the advantage of our neural
compression-based feature learning framework is greater when the noise
intensity is larger.

Qualitative Comparison. We show another two visual examples in Fig. 12 and
Fig. 13. As Fig. 12 shows, other methods cannot restore the bridge deck
texture well under this challenging case. The results of FastDVDNet and EMVD
suffer from serious distortion and are quite blurry. The results of EDVR,
BasciVSR, and BasicVSR++ show better visual quality but the bridge deck
textures are still not clear. By contrast, through leveraging better temporal
feature alignment and neural compression-based feature learning, our method
restores clearer and more accurate bridge deck textures. As Fig. 13 shows, the
grass patterns are smoothed in the results of FastDVDNet, EMVD, EDVR,
BasicVSR, and BasicVSR++. The results of our method are clearer and the
restored grass patterns are more similar to the target.

##  Appendix E More Comparisons on Video Deraining

We compare our method with prior SOTA video deraining methods, including MS-
CSC [21], SE [42], SpacCNN [12], FastDerain [19], J4RNet-P [25], FCRVD [46],
RMFD [47], and BasicVSR++ [9]. In the main paper, we present the results on
RainSynAll100 [47] and RainSynComplex25 [25]. In this appendix, we report the
results on NTU-Rain [12] and RainSynLight25 [25]. As Table 12 shows,
BasicVSR++ beats RMFD in terms of PSNR and SSIM on both testsets. Compared
with BasicVSR++, our method even brings PSNR gain of 0.74 dB and SSIM gain of
0.0035 on NTU-Rain, which shows the benefit of the noise-robust features.

We show visual comparison in Fig. 14. As Fig. 14 shows, BasicVSR++ could not
remove the rain streak well and suffers from severe artifacts. RMFD could
remove rain streak completely but its results suffer from serious color
shading. Instead, our method could produce clearer and visual pleasing
results.

##  Appendix F Visual Comparisons on Video Dehazing

Since the code of CG-IDN [50] is not released, we compare our method with
BasicVSR++[9] and MBSDN [14] in Fig. 15. As Fig. 15 shows, the result of
BasicVSR++ contains many artifacts and suffers from poor visual quality. The
result of MBSDN is cleaner but its color temperatures are inconsistent with
the target. Instead, our method could produce visual pleasing result with
accurate color temperatures.


# A Unified Hardware Architecture for Convolutions and Deconvolutions in CNN

Lin Bai, Yecheng Lyu and Xinming Huang  Worcester Polytechnic Institute
{lbai2, ylyu<EMAIL_ADDRESS>
###### Abstract

Deconvolution plays an important role in the state-of-the-art convolutional
neural networks (CNNs) for the tasks like semantic segmentation, image super
resolution, etc. In this paper, a scalable neural network hardware
architecture for image segmentation is proposed. By sharing the same computing
resources, both convolution and deconvolution operations are handled by the
same process element array. In addition, access to on-chip and off-chip
memories is optimized to alleviate the burden introduced by partial sum. As an
example, SegNet-Basic has been implemented using the proposed unified
architecture by targeting on Xilinx ZC706 FPGA, which achieves the performance
of 151.5 GOPS and 94.3 GOPS for convolution and deconvolution respectively.
This unified convolution/deconvolution design is applicable to other CNNs with
deconvolution.

##  I Introduction

In recent years, CNNs have been widely used in computer vision applications
such as classification [1], object detection [2], and semantic segmentation
[3]. However, a CNN usually requires intensive computations, which limits its
applicability on embedded devices. To address this issue, FPGA based
accelerators [4] have been proposed so that CNNs can be important on real-time
embedded systems. As a key operation in many neural networks, deconvolution
has been widely used in the state-of-the-art CNNs especially for semantic
segmentation[3], image super resolution[5], and image denoising[6]. Through a
learnable way, deconvolution extrapolates new information from input feature
maps, which outperforms other interpolation algorithms such as nearest
neighbor and bi-cubic interpolation. However, unlike hardware acceleration for
convolution, much less attention has been paid on deconvolution. Due to the
fact that deconvolution may become the bottleneck in speed if only convolution
has been accelerated, there is a urgent need to optimize the deconvolution
operation on FPGAs.

##  II Related Work

Lately tremendous research progress has been made on high-performance and low-
power CNN accelerators. In [7], the authors proposed a novel architecture for
process element array, which dramatically reduced the external memory
bandwidth requirements by intensive data reuse and outperforms the systolic-
like structures[8]. A high-throughput CNN accelerator design was implemented
in [9], where a comprehensive design space exploration on top of accurate
models was deployed to determine the optimal design configuration.

Comparing to the accelerator design for convolution, that of deconvolution has
not been thoroughly investigated. Liu et al. proposed an CNN architecture
where convolution and deconvolution were accelerated on the system
separately[10]. This architecture is not efficient enough because in most
CNNs, convolution and deconvolution do not work in parallel. A high
performance deconvolution module in [11] used reverse looping and stride hole
skipping techniques, but with the penalty of additional hardware resources and
latency. An unified systolic accelerator was developed in [12], which divided
the deconvolution into two steps. Firstly, it multiplied one vector with the
kernel and then stored the temporary matrices in on-chip memory. Next, it
added the overlaps of temporary matrices. This method increased the on-chip
BRAM access and introduced unnecessary data storage. Consequently both power
consumption and computation latency grew.

To address the issues mentioned above, we analyze the deconvolution properties
and fit it into our proposed process element array so that both convolution
and deconvolution can be handled by sharing the same on-chip resources. The
contributions of our work are summarized as follows:

  * •

A novel process element structure is proposed so that both convolution and
deconvolution are supported without extra circuit.

  * •

A scalable CNN accelerator architecture is proposed, in which data rate and
processing speed are configurable depending on the on-chip resources and
bandwidth of the target FPGA.

  * •

SegNet-Basic has been implemented on Xilinx ZC706 FPGA efficiently. Its
throughput are 151.5 GOPS and 94.3 GOPS for convolution and deconvolution
respectively, which outperforms most of the existing FPGA accelerators.

The rest of paper is organized as follows. Section III introduces the
background concept. The hybrid-optimization strategies are described in
Section IV. Hardware architecture and its implementation results of SegNet-
Basic are discussed in Section V-VII. In the end, Section VIII concludes the
paper.

##  III Backgrounds

###  III-A CNN architecture for segmentation

Typical segmentation neural network architecture consists of an encoder and a
decoder as shown in Fig. 1. The encoder is used to extract the features from
the input image and the decoder generates the segmented output. In CNNs like
SegNet and U-Net, decoder reverses their encoder, while others replace decoder
with smaller one. Additional connections between encoder and decoder were
introduced to SegNet [13] and U-Net [14]. Without the extra connection, CNN
still works but with a small decrease of precision. Usually, the encoder is
comprised of convolutional layers and the decoder consists of deconvolutional
layers.

Figure 1: Typical CNN structure for semantic segmentation

###  III-B Deconvolution

Deconvolution, also called transposed convolution, is a learnable method to
perform upsampling. If a convolution unit is directly reused for
deconvolution, it consists of the following two steps: 1) padding the input
feature map and 2) applying convolution on the padded feature map, as
indicated in Fig. 2. After padding, an input feature map with size
$I\\!F_{W}\times I\\!F_{H}$ is expanded into $(2\cdot
I\\!F_{W}+1)\times(2\cdot I\\!F_{H}+1)$, and consequently the output feature
map size becomes $(2\cdot I\\!F_{W}-1)\times(2\cdot I\\!F_{H}-1)$. In order to
get exactly twice the size, extra padding for upper row and left column
(highlighted in blue in Fig. 2) is needed.

Figure 2: Deconvolution operation by padding and convolution

##  IV Optimization

Because of the limited resources on an FPGA, a high performance CNN
accelerator has to be deeply optimized on memory access and data transfer
while maximizing the resource utilization.

###  IV-A Loop optimization

To efficiently map the convolution loops, three loop optimization techniques,
loop unrolling, loop tiling and loop interchange, have been considered to
customize the computation and communication patterns of the accelerator. Loop
unrolling is the parallelism strategy for certain convolution loops, which
demands more multipliers. Loop tiling determines the partition of feature
maps, and consequently determines the required size of on-chip memory. Loop
interchange decides the computation order of the convolution loops [15].

Figure 3: Four levels of convolution loops

After carefully judging all three optimization methods, our optimization
strategy is to unroll loop-1, and partially unroll loop-2, loop-4, and apply
loop tiling on depth of input feature maps (Fig. 3). In order to jointly
optimize with deconvolution, loop-1 is fully unrolled (more details about this
will be explained in the next section). In order to reduce the number of
partial sums and data transfer, loop-2 must be unrolled as much as possible.
However, as the large amount of multipliers are required, loop-2 is only
partially unrolled. Further consideration has been taken for on-chip memory
access minimization. Therefore, loop-4 is unrolled because of pixel reuse. As
the partial sum in loop-2 are stored in BRAM, no more overhead to the off-chip
memory is added.

###  IV-B Deconvolution

Fig. 2 presents the naive way to implement the deconvolution on hardware. As
it can be seen, too many operations are wasted in multiplication by zeros.

Figure 4: Optimization of deconvolution

The mathematical expression of the $2\times 2$ feature map deconvolution is
given in Fig. 4. Based on equation (1-4) for the deconvolution, we conclude
most of the redundant multiplications can be avoided. The procedure is
summarized into three steps: 1) padding the input feature map if size doubling
is expected; 2) scanning the padded input feature map by a $2\times 2$ sliding
window; 3) applying deconvolution for each patch using kernel as in equation
(1-4). Three examples are highlighted by colored squares in padded feature map
and output feature map in Fig. 4.

$\displaystyle O\\!F_{11}\\!=$ $\displaystyle
I\\!F_{11}\\!\cdot\\!K_{11}\\!+\\!I\\!F_{12}\\!\cdot\\!K_{13}\\!+\\!I\\!F_{21}\\!\cdot\\!K_{31}\\!+\\!I\\!F_{22}\\!\cdot\\!K_{33}$
(1) $\displaystyle O\\!F_{12}\\!=$ $\displaystyle
I\\!F_{12}\\!\cdot\\!K_{12}\\!+\\!I\\!F_{22}\\!\cdot\\!K_{32}$ (2)
$\displaystyle O\\!F_{21}\\!=$ $\displaystyle
I\\!F_{21}\\!\cdot\\!K_{21}\\!+\\!I\\!F_{22}\\!\cdot\\!K_{23}$ (3)
$\displaystyle O\\!F_{22}\\!=$ $\displaystyle I\\!F_{22}\\!\cdot\\!K_{22}$ (4)

According to TensorFlow, during deconvolution, the kernel should be rotated by
180°. To make the figure easier to understand, we assume that the kernel has
been rotated already.

###  IV-C Quantization method

A fine-tuning with quantization constraint method [16] is employed in our
design. It effectively diminishes the negative impact of brute-force
quantization while introducing more non-linearity. Different from the ordinary
quantization method [17], we quantize the weights and bias before storage.
This quantization method does not require modification of the TensorFlow
source code.

##  V Hardware Architecture

The overview of hardware architecture is shown in Fig. 5. Line buffer converts
the convolution into matrix multiplication by re-organizing the input image.
The process element array multiplies input image by the weights. After batch
normalization, activation and pooling, the output feature map is stored in
Output Featuremap (OF) buffer.

Figure 5: Overview of hardware architecture

###  V-A Line buffer

A line buffer is designed to build the expected sliding window and to perform
zero-padding for convolution. In the proposed accelerator, line buffer bridges
the AXI DMA and Input Featuremap (IF) buffer (Fig. 6). Data and valid signals
from AXI Stream interface are inserted into cascaded FIFOs. Extra logic is
added to generate the status signals (including empty and full signals) of
each FIFO, so that the line buffer is able to fit feature maps with different
sizes. Padding controller decides when to push the data into each FIFO and
when to output zero padding according to the pre-loaded padding mode.

Figure 6: Block diagram of line buffer

We choose not to buffer one entire feature map in on-chip memory, because the
buffer size would be restricted by the limited on-chip memory and this could
result inefficient buffer usage when feature map size drops. Therefore, only
part of the input feature map is loaded into IF buffer and consequently this
requires different zero-padding modes. Hence we provide different pre-loaded
work modes for this line buffer.

###  V-B Process element array

The input feature map is stored in the IF buffers in form of $3\times 1$
vector, which reduces the BRAM consumption. To rebuild the $3\times 3$ /
$2\times 2$ sliding window for convolution/deconvolution, a shift register is
placed between IF buffer and process element arrays. Each process element
array consists of multiple process elements as in Fig. 7, usually in a power
of 2. This number is scalable, depending on the bandwidth of platform. Each
process element comprises of a 9-multiplier array and an adder tree to sum up
the products.

Figure 7: PE array structure and its connection to buffers

####  V-B1 Convolution

During convolution, $3\times 3$ sliding windows and their corresponding
weights are transmitted into process element array. In the process element,
they are multiplied and summed up.

####  V-B2 Deconvolution

As discussed in Section IV-B, the deconvolution for each $2\times 2$ patch
consumes $9$ multiplications and $5$ additions. This fits well to the proposed
process element. During deconvolution, the process element structure is reused
but with a different data routing mechanism.

Different from convolution who outputs one pixel after another, in the
deconvolution mode, one process element generates $4$ pixels in parallel.
Considering the bitwidth of OF buffer, these output pixels are fed into buffer
in serial.

###  V-C Pooling

The pooling operation in CNNs is either max pooling or average pooling for
$2\times 2$ sliding windows. Another line buffer and shift register are
utilized to generate $2\times 2$ sliding windows. Its work mode can be
determined prior to compilation or configured on-the-fly.

###  V-D Batch normalization and activation

During inference, the batch normalization is downgraded into a multiplication
with an addition. We simply absorb it into the process element. Concerning to
the activation function, our design supports both ReLU and LeakyReLU.

###  V-E System controller

The system controller determines: 1) the data flow such as when to trigger
process element array, which data to be assigned into buffers, when to start
data transfer to DDR, and 2) the setting for each component mentioned above,
like the padding mode of line buffer module, work mode of process element
arrays, and whether to bypass activation or pooling modules. It is implemented
as a FSM. These settings are pre-defined and pre-loaded into register files so
that FSM only has to read and execute sequentially.

TABLE I: Resource comparison for CNN implementations on FPGA

[18] [19] [10] [11] Ours FPGA ZC7Z045 ZC7Z045 ZC7Z045 ZC7Z020 ZC7Z045 Clock
(MHz) 150 100 200 100 220 Precision 16-bit fixed 16-bit fixed 16-bit fixed
12-bit fixed 8-bit fixed Network VGG16-SVD VGG19 U-Net GAN SegNet-basic
Operation CONV CONV CONV+DECONV DECONV CONV+DECONV Performance 187 229 125
(CONV) 2.6 151.5 (CONV) (GOPS) 29 (DECONV) 94.3 (DECONV) Resource Efficiency
0.207(CONV) 0.254 0.14 (CONV) 0.012 0.168 (CONV) (GOPS/DSP) 0.033 (DECONV)
0.104 (DECONV)

##  VI Implementation Considerations of SegNet-Basic

The encoder part of SegNet-Basic includes $4$ convolution layers and $3$ max
pooling layers. The decoder part has $2$ convolution layers and $3$
deconvolution layers. The total parameter size for inference is about 42Mb,
with 8-bit quantization for feature maps and weights. The accelerator is
designed using Simulink and the HDL Coder toolbox. Our target platform Xilinx
ZC706 contains 900 DSP slices and 19.2Mb Block RAMs.

##  VII Results and Discussion

The test setup of SegNet-basic hardware accelerator is demonstrated in Fig. 8.
In the Zynq platform, hardware accelerator is loaded as a peripheral of ARM A9
processor. Two Direct Memory Access (DMAs) move the data between accelerator
and DDR memory. The input images and parameters are pre-loaded into memory and
transferred to PL by DMAs. This CNN accelerator clock frequency is 220MHz. Its
total resource consumption is summarized in Tab. II.

Figure 8: The hardware architecture of CNN accelerator TABLE II: Resource consumption LUTs | Registers | BRAMs | DSPs
---|---|---|---
16579 (8%) | 25390 (6%) | 537 (99%) | 576 (64%)

Comparing to other implementations (in Tab. I), our design achieves better
performance in case of deconvolution. Due to the sharing architecture, a
better balance on both performance and resource efficiency for convolution and
deconvolution is obtained. However, in order to support both operations, the
architecture is not deeply optimized for convolution specifically. Therefore,
the convolution performance is not as high as that from deeply optimized
implementation in [19].

###  VII-A Scalability

Scalability is represented by the number of process element arrays in the
accelerator. It is balance of bandwidth and computation capability. In SegNet-
Basic, the number of process element arrays is set to 1. This means the input
and output data bitwidth is 64. If higher bandwidth is supported, higher
performance is possible.

###  VII-B Latency of operations

In order to compare the latency, we perform convolution and max pooling on a
$90\times 120$ feature map (resulting a $45\times 60$ feature map) followed by
deconvolution. We find the time for convolution and deconvolution are the
same. The padding time difference is about $0.6\mu s$ due to different sizes
of input feature maps. Considering pooling and ReLU, another $1.2\mu s$ is
needed. Double buffering eliminates the data transfer time difference.
Therefore, deconvolution saves about 3.2% processing time if comparing to
convolution plus max-pooling and ReLU.

##  VIII Conclusions

In this paper, a scalable and configurable CNN accelerator architecture has
been proposed by combining both convolution and deconvolution into single
process element. The deconvolution operation is completed in one step and
buffering of intermediate results is not needed. In addition, SegNet-Basic has
been successfully implemented on Xilinx Zynq ZC706 FPGA that achieves the
performance of 151.5 GOPS for convolution and 94.3 GOPS for deconvolution,
which outperforms state-of-the-art segmentation CNN implementations.

## Acknowledgment

This work was supported by the Mathworks Inc.

## References

  * [1] K. Simonyan and A. Zisserman, ”Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [2] S. Ren, K. He, R. Girshick, and J. Sun, ”Faster R-CNN: Towards real-time object detection with region proposal networks,” In Advances in neural information processing systems (NIPS), pp. 91-99. 2015.
  * [3] J. Long, E. Shelhamer, and T. Darrell, ”Fully convolutional networks for semantic segmentation,” In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440. 2015.
  * [4] Y.H. Chen, J. Emer, and V. Sze, ”Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” In Proceedings of the 43rd International Symposium on Computer Architecture (ISCA), pp. 367-379, 2016.
  * [5] C. Ledig, et al, ”Photo-realistic single image super-resolution using a generative adversarial network,” In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681-4690. 2017.
  * [6] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, ”Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142-3155, 2017.
  * [7] U. Aydonat, S. O’Connell, D. Capalija, A.C. Ling, and G.R. Chiu, ”An opencl™ deep learning accelerator on arria 10,” In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 55-64. 2017.
  * [8] C. Zhang, Z. Fang, P. Zhou, P. Pan, J. Cong, ”Caffeine: towards uniformed representation and acceleration for deep convolutional neural networks,” In Proceedings of the 35th International Conference on Computer-Aided Design, p. 12, 2016.
  * [9] X. Wei, C.H. Yu, P. Zhang, Y. Chen, Y. Wang, H. Hu, Y. Liang, and J. Cong, ”Automated systolic array architecture synthesis for high throughput CNN inference on FPGAs,” In Proceedings of the 54th Annual Design Automation Conference, p. 29, 2017.
  * [10] S. Liu, H. Fan, X. Niu, H. Ng, Y. Chu, and W. Luk, ”Optimizing CNN-based Segmentation with Deeply Customized Convolutional and Deconvolutional Architectures on FPGA,” ACM Transactions on Reconfigurable Technology and Systems (TRETS), vol. 11, no. 3, 2018.
  * [11] X. Zhang, S. Das, O. Neopane, K. Kreutz-Delgado, ”A Design Methodology for Efficient Implementation of Deconvolutional Neural Networks on an FPGA,” arXiv preprint arXiv:1705.02583, 2017.
  * [12] D. Xu, K. Tu, Y. Wang, C. Liu, B. He, and H. Li, ”FCN-element: accelerating deconvolutional layers in classic CNN processors,” In Proceedings of the International Conference on Computer-Aided Design, pp. 22, 2018.
  * [13] V. Badrinarayanan, A. Kendall, and R. Cipolla, ”Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481-2495, 2017.
  * [14] O. Ronneberger, P. Fischer, and T. Brox, ”U-net: Convolutional networks for biomedical image segmentation,” In International Conference on Medical Image Computing and Computer Assisted Intervention, pp. 234-241, 2015.
  * [15] Y. Ma, Y. Cao, S. Vrudhula, and J. Seo, ”Optimizing loop operation and dataflow in FPGA acceleration of deep convolutional neural networks,” In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 45-54, 2017.
  * [16] Y. Lyu, L. Bai, and X. Huang., ”Chipnet: Real-time LiDAR processing for drivable region segmentation on an FPGA,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 66, no. 5, pp. 1769 - 1779, 2019.
  * [17] B. Jacob, S. Kligys, B. Chen, M. Zhu, M.Tang, A. Howard, H. Adam, and D. Kalenichenko, ”Quantization and training of neural networks for efficient integer-arithmetic-only inference,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2704-2713, 2018.
  * [18] J. Qiu, J. Wang, S. Yao et al, ”Going deeper with embedded fpga platform for convolutional neural network,” In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays(FPGA), pp. 26-35, 2016.
  * [19] Q. Xiao, Y. Liang et al, ”Exploring heterogeneous algorithms for accelerating deep convolutional neural networks on FPGAs,” In 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC), pp. 1-6, 2017.