# Multi-Outputs Is All You Need For Deblur

Sidun Liu, Peng Qiao111Peng Qiao is the corresponding author, Yong Dou

###### Abstract

Image deblurring task is an ill-posed one, where exists infinite feasible
solutions for blurry image. Modern deep learning approaches usually discard
the learning of blur kernels and directly employ end-to-end supervised
learning. Popular deblurring datasets define the label as one of the feasible
solutions. However, we argue that it’s not reasonable to specify a label
directly, especially when the label is sampled from a random distribution.
Therefore, we propose to make the network learn the distribution of feasible
solutions, and design based on this consideration a novel multi-head output
architecture and corresponding loss function for distribution learning. Our
approach enables the model to output multiple feasible solutions to
approximate the target distribution. We further propose a novel parameter
multiplexing method that reduces the number of parameters and computational
effort while improving performance. We evaluated our approach on multiple
image-deblur models, including the current state-of-the-art NAFNet. The
improvement of best overall (pick the highest score among multiple heads for
each validation image) PSNR outperforms the compared baselines up to
0.11$\sim$0.18dB. The improvement of the best single head (pick the best-
performed head among multiple heads on validation set) PSNR outperforms the
compared baselines up to 0.04$\sim$0.08dB. The codes are available at
https://github.com/Liu-SD/multi-output-deblur.

## 1 Introduction

Image restoration, aiming at reconstructing a high-quality image from its
degraded counterpart, is one of the fundamental problems in computer vision.
Image deblurring is one of these tasks, where degradations are caused by the
motion or jittering of camera or objects. The task of image deblurring is to
remove the blurry trace and restore the sharp image. When the exposure time is
not short enough, the image records the instantaneous movement of the objects,
resulting in the blurry one.

With the help of the powerful fitting ability of CNN-based and recently
popular self-attention-based (Vaswani et al. 2017; Dosovitskiy et al. 2020)
models, the supervised image deblurring achieves tremendous improvement (Dong
et al. 2014; Qiao et al. 2017; Chen et al. 2021a; Liang et al. 2021; Chen et
al. 2021b; Park et al. 2020; Zamir et al. 2022; Wang et al. 2022; Chu et al.
2021). MPRNet (Zamir et al. 2021) uses a multi-stage training strategy to
progressively restore a degraded image; HiNet (Chen et al. 2021b) uses a Half
instance normalization layer to boost the restoration; TLC (Chu et al. 2021)
restricts the statistics aggregation to local at test phase; Restormer (Zamir
et al. 2022) and MAXIM (Tu et al. 2022) put the self-attention to feature
dimension to relief the computing complexity; NAFNet (Chen et al. 2022)
combines the MobileNet-style (Howard et al. 2017) convolution module and
channel attention (Hu, Shen, and Sun 2018) to make the model lightweight but
powerful. The sophisticated architecture of these networks gives the models
strong representation ability, allowing them to excel at the deblurring task.
In supervised learning, we consider a blurry image as a certain stacking of a
frame sequence, the corresponding label image is usually defined as the middle
frame in this sequence. However, due to the random jittering of camera or
objects during the image capturing, the middle frame is still with
uncertainty. There may exist several pixel shifts between the label image and
the restored image. When training models with respect to MSE-like loss
function, the restored images lose the sharp edges and meaningful textures.

Figure 1: The training flow of the proposed multi-outputs approach. The deep
neural network generates multiple sharp outputs. But only the output with the
lowest loss is used for back-propagation.

In a supervised deblurring task, we assume that the corresponding label images
are samples from a distribution but not ones defined above. The samples in
this distribution are all feasible restoration results. Training the models
using the MSE-like loss function can only capture the expectation of this
distribution, which limits the quality of the restored images.

In this scenario, it is better to train the model to learn the distribution,
instead of learning its expectation. However, learning the distribution
directly is difficult (Kingma and Welling 2013). We propose to divide the
distribution into several clusters, supervising the model to learn the
expectations of each cluster. Inspired by k-means (Lloyd 1982), we incorporate
an EM algorithm to optimize the model to capture the distribution. At the
beginning, the cluster centers are randomly initialized. For E-step, each
label is assigned to the nearest cluster, and for M-step, the cluster centers
are updated to be the expectations of their corresponding clusters. For
implementation, we design a multi-head output layer to generate clustering
centers. As only the output layer and supervision manner are modified, our
proposed method can be easily extended to existing models.

The more clusters, the better the distribution is fitted. But directly
increasing the number of output heads is not acceptable. On the one hand, the
parameters and computations increase linearly as the number of clusters
increasing. On the other hand, some heads may not be sufficiently utilized. We
observe that there are correlations between the output images from different
heads. As described above, the transformation from sharp images to a blurry
image is additive. Therefore, we combine the heads of model output in pairs to
obtain the extended multi-head output. In this way, the parameters and
computations of the last layer are the square roots of the number of clusters.
Meanwhile, the shared parameters will enhance the connections between
different heads.

The experiments are performed on multiple models, including the NAFNet (Chen
et al. 2022), Restormer (Zamir et al. 2022), HiNet (Chen et al. 2021b), and
MPRNet (Zamir et al. 2021), trained on GoPro (Nah, Kim, and Lee 2017) and
validated on various datasets. The experiments on NAFNet-width32 (Chen et al.
2022) show that with our multi-head extension, the best overall (pick the
highest score among multiple heads for each validation image) PSNR improves
0.05$\sim$0.22dB with a different number of heads, and the best single head
(pick the best-performed head among multiple heads on validation set) PSNR
outperforms it up to 0.2dB. When the head combination is enabled, the
experimented models achieve 0.15dB higher best overall PSNR and 0.05dB higher
best single head PSNR on average, compared to single head baselines of the
models above. The multi-head NAFNet-width64 with head combination achieves
33.82 (+0.11dB) best overall PSNR, and 33.75 (+0.04dB) best single head PSNR,
which exceeds the state-of-the-art.

We further analyze the multi-head outputs and find that with our semi-
supervised training strategy, multiple feasible but different results are
generated. The diversity of the multi-headed outputs are reflected in the
blurry regions of the image.

Our contributions can be summarized as follows:

  * •

We analyze the reason that limits image recovery performance in supervised
image deblurring tasks. Since deblurring is ill-posed, using labeled image
supervision will likely result in the model outputting images out of the
distribution.

  * •

We point out that the distribution of sharp images should be learned, not
their expectations. And a semi-supervised learning strategy based on the EM
algorithm is proposed to learn multiple clusters to approximate the
distribution.

  * •

We further propose to combine the output heads in pairs such that the
parameters and computations of the output layer is the square root of the
number of clusters. The connection between the clusters is also enhanced by
sharing parameters.

  * •

Our approach can be simply extended to existing models. We use NAFNet and
other mainstream models as backbones for our experiments. The experiments
demonstrate the effectiveness of the proposed method. Not only does the best
overall PSNR exceed the baselines, but also the PSNR of each head is
comparable to or better than them.

## 2 Related Work

### 2.1 Image Deblurring

The deep learning methods have achieved significant success in image
deblurring and other low-level vision tasks such as image denoise (Tian et al.
2020), image deraining (Li et al. 2019), and image super-resolution (Yang et
al. 2019). Early works propose to estimate the blur kernel (Chakrabarti 2016;
Ren et al. 2020; Schuler et al. 2015; Sun et al. 2015; Tran et al. 2021). But,
since the characteristics of blur are complex, the blur kernel estimation
method is not practical in real scenarios. DeepDeblur (Nah, Kim, and Lee 2017)
gives up estimating blur kernel but directly maps a blurry image to its sharp
counterpart. Following this paradigm, a series of methods (Zamir et al. 2021;
Chen et al. 2021b; Zamir et al. 2022; Chu et al. 2021) constantly refresh the
SOTA. At present, the NAFNet (Chen et al. 2022) achieves highest PSNR (33.71
dB) on the GoPro (Nah, Kim, and Lee 2017) image deblur dataset.

### 2.2 Deblurring Datasets

The commonly used benchmark for image deblurring is GoPro (Nah, Kim, and Lee
2017), where the input blurry images are synthetic. The GoPro dataset takes
240 fps videos with a GOPRO camera and then averages varying numbers (7 - 13)
of successive latent frames to produce blurs of different strengths. After
averaging, the integrated signal is then transformed into pixel value by
nonlinear CRF (Camera Response Function).

HIDE (Shen et al. 2019) is a dataset for human-ware image deblurring, where
the images consist of densely annotated foreground human bounding boxes. The
blurry images from the HIDE are synthetic in the same way as GoPro.

The ill-posed nature of deblurring makes every sharp image corresponding to
each blurry one a feasible solution. For the convenience of supervision, both
datasets define the middle frame among the sharp frames as the corresponding
sharp image. However, due to the random jittering of camera or objects during
the image capturing, the middle frame is stochastic, forming a distribution.
Such randomness makes the model impossible to predict the exact middle frame
for the input blurry image.

### 2.3 Distribution Learning

To learn the distribution, we have to assume it belongs to a restricted family
of distributions. Gaussian mixture model (GMM) (Reynolds 2009) is a
distribution estimation model with an assumption that the data are sampled
from multiple Gaussian distributions. An EM algorithm exists to estimate the
expectation and covariance matrix for the GMM. A special case is derived when
the hard partition is made. Then the GMM degrades to the k-means (Lloyd 1982)
problem.

In the deblurring task, the restoration result follows an unknown distribution
$P(y|x)$. When the distribution is approximated by the mixtures of Gaussian
distribution with hard partition, the k-means-like algorithm can be applied to
learn the distribution.

## 3 Approach

In this section, we provide more detailed explanations about the multi-head
training strategy and the head combination method in the following
subsections.

### 3.1 Multi-Head Training

Image restoration, e.g. deblurring, is an ill-posed problem as there exists
infinite feasible solutions. We expect the learned model to generate one of
these feasible solutions. However, when the supervised training is applied,
the model is forced to generate the one which exactly matches the target sharp
image. If the selection of the target sharp image is deterministic, the model
is able to learn to generate the restored image. However, in fact, the
selection of target sharp images is stochastic. E.g., the popular single image
deblurring dataset GoPro (Nah, Kim, and Lee 2017) synthesizes the input blurry
image with a picture sequence and selects the middle frame of that sequence as
the label. The random jittering of camera or objects during the capturing
still causes the randomness to the target sharp image. If we assume the label
is sampled from an unknown distribution, the model trained in a supervised
manner is tend to generate the expectation of such distribution. However, the
expectation solution may provide inferior visual quality. In fact, for the
motion deblurring task, there exist several pixel shifts between feasible
solutions. So the expectation of such distribution loses sharp edges and
meaningful textures, making it far from feasible solutions.

To tackle this problem, we point out that the distribution of the target sharp
image should be learned, instead of its expectation. However, without any
prior about the distribution, learning the distribution directly is difficult.
Therefore, we divide the distribution into several clusters, represented by
their cluster centers. When the cluster number is large enough, the
distribution can be approximated by these centers. Then we supervise the model
to generate these cluster centers.

Given a set of image pairs $\\{(x_{i},y_{i})\\}_{i=1}^{n}$, the model
generates $K$ cluster centres $\\{\mu_{j}(x)\\}_{j=1}^{K}$ for each input
image $x$. We aim to assign the label $y$ to one of these clusters, so as to
minimize the within-cluster sum of squared errors. Formally, the objective is
to find assignment $S$ and cluster centers generator $\mu$ as Equation 1.

$\operatorname*{arg\,min}_{\mathbf{S},\mu}\sum_{i=1}^{K}\sum_{(x,y)\in\mathbf{S}_{i}}||y-\mu_{i}(x)||^{2}$
(1)

This problem is NP-hard. Inspired by k-means, we designed an EM algorithm to
heuristically optimize this objective in Equation 1. Given an initial cluster
centers generator $\mu^{(1)}$, the algorithm proceeds by alternating between
two steps:

Assignment step: Assign each label $y$ to the cluster with the nearest cluster
centres (Equation 2).

$\mathbf{S}_{i}^{(t)}=\\{(x,y):||y-\mu^{(t)}_{i}(x)||^{2}\leq||y-\mu^{(t)}_{j}(x)||^{2}\\\
\forall j,1\leq j\leq K\\}$ (2)

Update step: Update each cluster centre generator $\mu^{(t)}$ with its
assigned image pairs in ${S}^{(t)}$.

$\mu_{i}^{(t+1)}=\min\sum_{(x,y)\in\mathbf{S}_{i}^{(t)}}||y-\mu_{i}^{(t)}(x)||^{2}$
(3)

In practice, the deep neural network (DNN) with multi-head outputs is used as
a generator $\mu$. Each head $\mu_{i}$ is parameterized by $\theta_{i}$. The
assignment and update step with DNN is transformed into:

Assignment step with DNN: Find the head $\mu_{\theta_{\hat{k}}}$ with the
minimum loss $L$ with respect to the label $y$.

$\hat{k}=\operatorname*{arg\,min}_{i}L(\mu_{\theta_{i}}(x),y)\quad 1\leq i\leq
K$ (4)

$L$ is the MSE-like loss, e.g. Charbonnier loss (Charbonnier et al. 1994) or
PSNR loss.

Update step with DNN: Update the parameter $\theta_{\hat{k}}$ of head
$\mu_{\theta_{\hat{k}}}$ to get one-step closer to $y$.

$\mu_{\theta_{\hat{k}}}=\mu_{\theta_{\hat{k}}}-\alpha\nabla
L(\mu_{\theta_{\hat{k}}}(x),y)$ (5)

$\alpha$ is the learning rate.

Figure 2: An example of head combination method when the number of heads is 3.
The multi-head outputs are combined in pairs (including themselves) to obtain
the extended outputs.

### 3.2 Head Combination

Figure 3: A heatmap visualization to show the residuals between different
heads. A trained 4-head NAFNet-width32 with the head combination is used for
illustration. The first row is the sharp image, and the second row is the
corresponding blurry image. The heatmaps in the third row are computed by
accumulating and normalizing the absolute residuals for all pairs of extended
heads. Differences between outputs are mainly in the blurry regions,
especially at their edges. Figure 4: An visualization to show the pixel shifts
between different heads. A trained 4-head NAFNet-width64 is used for
illustration. For the difference between head $i$ and $j$, if the pixel value
of $i$ is larger than that of $j$, the corresponding position becomes bright,
and vice versa. An object’s left edge is bright and right edge is dark means
that this object shifts left from output $i$ to output $j$.

In the previous section, we described the proposed approach to approximate the
distribution of target sharp images with multiple cluster centers. In general,
more cluster centers may lead to greater performance gains. However, in
practice, this causes some problems. First, the parameters and computations of
the output layer grow linearly to the number of heads. Second, as we only
update the nearest head to label, the heads with relatively worse performance
have less chance to be updated, which means some heads can’t be sufficiently
utilized.

We notice as in Figure 3 that there are correlations between the multi-head
outputs. Some slight shifts occur between heads. We assume the input blurry
image is a certain stack of a frame sequence, then the processing done by each
head is to erase with a residual network architecture all frames but the frame
to be restored. So the functions of these heads overlap to some extent. These
overlaps can be encoded by sets of shared parameters, and the parameters and
computations can be reduced.

Based on this assumption, we proposed a head combination method. We traverse
all the output heads and combine them in pairs (including themselves) to get
the extended multi-head outputs. Then we apply the multi-head loss to them. As
the transformation from sharp to a blurry image is additive, we use simple
addition as the combination strategy. The head combination is described as
Equation 6.

$\mu^{\prime}_{i,j}(x)=\frac{\mu_{i}(x)+\mu_{j}(x)}{2}\quad 1\leq i\leq j\leq
K$ (6)

If the model has $K$ output heads, then the number of extended heads is
$K^{\prime}=K(K+1)/2$.

With the head combination, the parameters and computations of the last layer
are square-root to the number of extended heads. Besides, the correlations
between heads are enhanced by sharing parameters. When one of the extended
heads is updated, other correlated heads are updated as well, which avoids the
problem of insufficient training for certain heads.

## 4 Experiments

### 4.1 Dataset

We use GoPro (Nah, Kim, and Lee 2017) datasets for evaluation. GoPro is a
commonly used motion deblurring dataset that contains 2,103 image pairs for
training and 1,111 pairs for evaluation.

Furthermore, to demonstrate generalizability (Zamir et al. 2021; Wang et al.
2022; Chen et al. 2022), we take the GoPro trained model and directly apply it
on the test images of HIDE (Shen et al. 2019) dataset. The HIDE’s test set
contains 2,025 images, specifically built for human-aware deblurring.

### 4.2 Implementation Details

We use multiple commonly compared models to evaluate our method, including:

MPRNet (Zamir et al. 2021) is a multi-stage architecture, which progressively
learns restoration functions for the degraded inputs, thereby breaking down
the overall recovery process into several steps. As there are multiple outputs
from different steps, we adopt the multi-head output layer to all output
layers.

HINet (Chen et al. 2021b) uses a novel Half Instance Normalization Block (HIN
Block) to boost the performance of the image restoration network.

Restormer (Zamir et al. 2022) propose a multi-Dconv head transposed attention
(MDTA) module to aggregate local and non-local pixel interactions, which is
efficient to process high-resolution images.

NAFNet (Chen et al. 2022) use a simple U-shaped architecture to achieve the
SOTA. There are only MobileNet-style (Howard et al. 2017) convolutions, SE
(Hu, Shen, and Sun 2018) channel attentions, and shortcuts on the NAFNet.

For all the models above, we replace the original output layer with our multi-
head output layer by expanding the channels of the output layer by a factor of
$K$. The hyper-parameters and settings are kept the same as reported in the
papers.

We use the base version of NAFNet with latent width of 32 (NAFNet-width32) for
ablation study and analysis. All the models are used for overall evaluation.
All the experiments are conducted on a server with 8 Tesla V100 GPUs.

### 4.3 Ablation Study

(a) Without the head combination.

(b) With the head combination.

Figure 5: Ablation study on the influence of the number of heads to two types
of metrics defined in equation 7a and 7b. Besides, the PSNRs for all heads are
plotted with blue $\blacktriangle$.

In this section, we train our multi-head NAFNet-width32 on the GoPro dataset
with a variant number of heads. The impact of head combination is also
evaluated. There are two types of metrics we used to evaluate our method. The
first is the $P\\!S\\!N\\!R_{overall}$, which picks the highest score among
multiple heads for each validation image (Equation 7a); the second is the
$P\\!S\\!N\\!R_{single\mbox{-}head}$, which picks the best performed head
among multiple heads on validation set (Equation 7b). Here, the (extended)
head number is $K$, and the size of validation set is $N$.

$\displaystyle
P\\!S\\!N\\!R_{overall}=\frac{1}{N}\sum_{i=1}^{N}\max_{j}P\\!S\\!N\\!R\left(\mu_{j}(x_{i}),y_{i}\right)$
(7a) $\displaystyle
P\\!S\\!N\\!R_{single\mbox{-}head}=\max_{j}\frac{1}{N}\sum_{i=1}^{N}P\\!S\\!N\\!R\left(\mu_{j}(x_{i}),y_{i}\right)$
(7b) $\displaystyle 1\leq j\leq K$

We show the effects of head numbers on model performance in figure 5(a). It is
observed that not only the $P\\!S\\!N\\!R_{overall}$ but also the
$P\\!S\\!N\\!R_{single\mbox{-}head}$ is positively correlated with the number
of heads. However, the parameter numbers and computation cost limit the
further increment of head numbers. Not all heads perform equally well. A
possible explanation is that some heads are in charge of recovering out-of-
distribution samples, while other heads are not sufficiently trained.

Figure 5(b) shows the PSNR gains when the head combination is enabled. We can
observe that the 4-head model with combination performs better than the 8-head
model without combination. On the one hand, a 4-head combination is able to
generate 10 outputs. On the other hand, the shared parameters encoded the
connections between heads, so that more heads are sufficiently utilized.
However, the gains of the $P\\!S\\!N\\!R_{overall}$ slow down and the
$P\\!S\\!N\\!R_{single\mbox{-}head}$ drops when the number of heads goes to 5.
So we set the head numbers to be 4 and enable the head combination for
subsequent experiments.

### 4.4 Analysis on Multi-Head Architecture

(a) $\\#heads=4$

(b) $\\#heads=5$

Figure 6: The PSNR of the output from each pair of heads, using the heatmap
format.

In this section, we analyze the restored sharp images from multiple heads and
answer the following questions.

How do multi-head outputs look like? The differences between the output images
are barely observable with human eyes, but the pixel-wise differences are
significant. The visualization in figure 3 shows that the differences between
heads are mainly in the blurry regions, especially the edge of objects. For
these blurry regions, there exist infinite solutions, and our trained model
generates multiple different solutions. So the heatmap is lighter in the
corresponding area. The visualization proves that our trained model is able to
generate multiple feasible solutions for the blurry regions, instead of their
expectation. One of the outputs matches the label most, but all of them are
feasible as a sharp image.

If the camera shakes when taking pictures, the entire image is blurry, so the
pixel shift of the entire image should be observed when multi-head output is
applied. We use a trained 4-head model without head combination to visualize
such a phenomenon in the 2nd picture of figure 3, where the camera shakes
violently. The figure shows the normalized residuals between each pair of
outputs. A brighter area means the pixel value increase and vice versa. E.g.,
an object’s left edge is bright and the right edge is dark means that this
object shifts to the left. We can observe that the edge of all the objects in
the picture have similar patterns, proving that there exist pixel shifts
between multi-head outputs.

Table 1: The results on various motion deblurring datasets. The models with the suffix -MH-C are our multi-head variants with the head combination. The models are trained only on the GoPro dataset and directly applied to the HIDE benchmark dataset. ∗Single head for $P\\!S\\!N\\!R_{single\mbox{-}head}/S\\!S\\!I\\!M_{single\mbox{-}head}$ and Overall for $P\\!S\\!N\\!R_{overall}/S\\!S\\!I\\!M_{overall}$. Method | Metric∗ | GoPro | HIDE
---|---|---|---
PSNR | SSIM | PSNR | SSIM
Gao et al. (Gao et al. 2019) | - | 30.90 | 0.935 | 29.11 | 0.913
DBGAN (Zhang et al. 2020) | - | 31.10 | 0.942 | 28.94 | 0.915
MT-RNN (Park et al. 2020) | - | 31.15 | 0.945 | 29.15 | 0.918
DMPHN (Zhang et al. 2019) | - | 31.20 | 0.940 | 29.09 | 0.924
Suin et al. (Suin et al. 2020) | - | 31.85 | 0.948 | 29.98 | 0.930
SPAIR (Purohit et al. 2021) | - | 32.06 | 0.953 | 30.29 | 0.931
MIMO-UNet+ (Cho et al. 2021) | - | 32.45 | 0.957 | 29.99 | 0.930
IPT (Chen et al. 2021a) | - | 32.52 |  - |  - |  -
MAXIM-3S (Tu et al. 2022) | - | 32.86 | 0.961 | 32.83 | 0.956
MPRNet (Zamir et al. 2021) | - | 32.66 | 0.959 | 30.96 | 0.939
MPRNet-MH-C (Ours) | Single head | 32.65-0.01 | 0.959+0.00 | 30.95-0.01 | 0.940+0.01
Overall | 32.78+0.12 | 0.960+0.01 | 31.07+0.11 | 0.940+0.01
HINet (Chen et al. 2021b) | - | 32.77 | 0.959 | 30.33 | 0.932
HINet-MH-C (Ours) | Single head | 32.83+0.06 | 0.960+0.01 | 30.41+0.08 | 0.933+0.01
Overall | 32.95+0.18 | 0.961+0.02 | 30.50+0.17 | 0.934+0.02
Restormer (Zamir et al. 2022) | - | 32.92 | 0.961 | 31.22 | 0.942
Restormer-MH-C (Ours) | Single head | 33.01+0.09 | 0.962+0.01 | 31.36+0.14 | 0.944+0.02
Overall | 33.11+0.19 | 0.963+0.02 | 31.41+0.19 | 0.945+0.03
NAFNet (Chen et al. 2022) | - | 33.71 | 0.966 | 31.22 | 0.943
NAFNet-MH-C (Ours) | Single head | 33.75+0.04 | 0.967+0.01 | 31.28+0.06 | 0.944+0.01
Overall | 33.82+0.11 | 0.967+0.01 | 31.33+0.11 | 0.944+0.01

How does head combination influence the results? The results in section 4.3
show that head combination improves the model performance while halving the
parameters and computations of the output layer. Figure 6 plots the PSNR
scores of all pairs of heads for the 4-head and 5-head model respectively. We
can observe that the single head output on the diagonal performs worse, while
the pair with different heads performs better. A possible explanation is that
combining two different heads aggregates more information than a single head.
The maximization operation allows for positive feedback during model training,
which in turn results in better performance for the combination of different
heads and worse performance for the single head. We further observe that There
is no intersection between the best-performed pair of heads, i.e. pair
$h0\&h1$ and $h2\&h3$. For $h4$ in the 5-head model, there is no other head to
form a pair with it, so it performs moderately well with all other heads.

Why do multi-head models achieve better $P\\!S\\!N\\!R_{single\mbox{-}head}$
than their single-head counterparts? We hypothesize that in deblur datasets
like GoPro, the majority of labels are exactly the middle frame, and are
deterministic when blurry input is given. But other parts of labels shift
relative to the actual middle frames. When the single-output network is
trained to fit such pairs, large losses occur, which may have a negative
impact on training. However, the shifted label is still one feasible solution.
Therefore, the proposed multi-outputs model allows a portion of the outputs in
charge of restoring such pairs, making the model benefit from these samples
while avoiding the negative effects of shifts on the output layer. So the
multi-head models have a more stable training process and achieve better
$P\\!S\\!N\\!R_{single\mbox{-}head}$.

### 4.5 Motion Deblurring Results

In this section, we train our multi-outputs model on GoPro and evaluate it on
GoPro and HIDE datasets. Table 1 shows the results compared to the baseline
single-output model and other approaches. The multi-head model with the head
combination is represented with suffix -MH-C.

In details, the PSNR on GoPro of MPRNet, HINet, Restormer and NAFNet are
improved by -0.01dB, 0.06dB, 0.08dB, 0.04dB, respectively. The PSNR on HIDE of
MPRNet, HINet, Restormer and NAFNet are improved by -0.01dB, 0.08dB, 0.14dB,
0.06dB, respectively. The three-stage output scheme of MPRNet limits gains of
its $P\\!S\\!N\\!R_{single\mbox{-}head}$. The metric $P\\!S\\!N\\!R_{overall}$
achieves 0.1dB gains on average, which means that our multi-outputs model is
able to generate sharper images. When one of the generated sharp images
matches the label, the PSNR gets higher.

## 5 Conclusion

Image deblurring is an ill-posed problem, and there are many feasible
solutions for a blurry image. In this paper, we propose a multi-head outputs
method to learn the distribution of these solutions. The method can be easily
adapted to existing image restoration models. With the proposed multi-head
outputs extension, as well as the corresponding loss function, the model is
able to learn the distribution of sharp images given blurry input, instead of
its expectation. In particular, the feasible solutions are aggregated into
multiple clusters, and the objective of the training is to minimize the
within-cluster sum of squared error. For implementation, the model outputs
multiple sharp images, and only the head with the smallest loss is used for
back-propagation. To further improve the utilization of each head, we propose
a head combination method. The outputs are combined in pairs to obtain the
extended outputs. The experiments on multiple models and datasets show that
not only does the best overall PSNR metric exceed the baselines, but also the
PSNR of each head is comparable to or even better than them. By extending the
NAFNet, the proposed method surpasses the SOTA on the GoPro dataset.

## References

  * Chakrabarti (2016) Chakrabarti, A. 2016.  A neural approach to blind motion deblurring.  In _European conference on computer vision_ , 221–235. Springer.
  * Charbonnier et al. (1994) Charbonnier, P.; Blanc-Feraud, L.; Aubert, G.; and Barlaud, M. 1994.  Two deterministic half-quadratic regularization algorithms for computed imaging.  In _Proceedings of 1st International Conference on Image Processing_ , volume 2, 168–172. IEEE.
  * Chen et al. (2021a) Chen, H.; Wang, Y.; Guo, T.; Xu, C.; Deng, Y.; Liu, Z.; Ma, S.; Xu, C.; Xu, C.; and Gao, W. 2021a.  Pre-trained image processing transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12299–12310.
  * Chen et al. (2022) Chen, L.; Chu, X.; Zhang, X.; and Sun, J. 2022.  Simple baselines for image restoration.  _arXiv preprint arXiv:2204.04676_.
  * Chen et al. (2021b) Chen, L.; Lu, X.; Zhang, J.; Chu, X.; and Chen, C. 2021b.  HINet: Half instance normalization network for image restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 182–192.
  * Cho et al. (2021) Cho, S.-J.; Ji, S.-W.; Hong, J.-P.; Jung, S.-W.; and Ko, S.-J. 2021.  Rethinking coarse-to-fine approach in single image deblurring.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 4641–4650.
  * Chu et al. (2021) Chu, X.; Chen, L.; ; Chen, C.; and Lu, X. 2021.  Improving Image Restoration by Revisiting Global Information Aggregation.  _arXiv preprint arXiv:2112.04491_.
  * Dong et al. (2014) Dong, C.; Loy, C. C.; He, K.; and Tang, X. 2014.  Learning a deep convolutional network for image super-resolution.  In _European conference on computer vision_ , 184–199. Springer.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Gao et al. (2019) Gao, H.; Tao, X.; Shen, X.; and Jia, J. 2019.  Dynamic scene deblurring with parameter selective sharing and nested skip connections.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 3848–3856.
  * Howard et al. (2017) Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_.
  * Hu, Shen, and Sun (2018) Hu, J.; Shen, L.; and Sun, G. 2018.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7132–7141.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Li et al. (2019) Li, S.; Araujo, I. B.; Ren, W.; Wang, Z.; Tokuda, E. K.; Junior, R. H.; Cesar-Junior, R.; Zhang, J.; Guo, X.; and Cao, X. 2019.  Single image deraining: A comprehensive benchmark analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3838–3847.
  * Liang et al. (2021) Liang, J.; Cao, J.; Sun, G.; Zhang, K.; Van Gool, L.; and Timofte, R. 2021.  Swinir: Image restoration using swin transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 1833–1844.
  * Lloyd (1982) Lloyd, S. 1982.  Least squares quantization in PCM.  _IEEE transactions on information theory_ , 28(2): 129–137.
  * Nah, Kim, and Lee (2017) Nah, S.; Kim, T. H.; and Lee, K. M. 2017.  Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring.  In _CVPR_.
  * Park et al. (2020) Park, D.; Kang, D. U.; Kim, J.; and Chun, S. Y. 2020.  Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training.  In _European Conference on Computer Vision_ , 327–343. Springer.
  * Purohit et al. (2021) Purohit, K.; Suin, M.; Rajagopalan, A.; and Boddeti, V. N. 2021.  Spatially-adaptive image restoration using distortion-guided networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2309–2319.
  * Qiao et al. (2017) Qiao, P.; Dou, Y.; Feng, W.; Li, R.; and Chen, Y. 2017.  Learning Non-Local Image Diffusion for Image Denoising.  In _Proceedings of the 25th ACM International Conference on Multimedia_ , 1847–1855.
  * Ren et al. (2020) Ren, D.; Zhang, K.; Wang, Q.; Hu, Q.; and Zuo, W. 2020.  Neural blind deconvolution using deep priors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3341–3350.
  * Reynolds (2009) Reynolds, D. 2009.  _Gaussian Mixture Models_ , 659–663.  Springer US.
  * Schuler et al. (2015) Schuler, C. J.; Hirsch, M.; Harmeling, S.; and Schölkopf, B. 2015.  Learning to deblur.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(7): 1439–1451.
  * Shen et al. (2019) Shen, Z.; Wang, W.; Lu, X.; Shen, J.; Ling, H.; Xu, T.; and Shao, L. 2019.  Human-aware motion deblurring.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5572–5581.
  * Suin et al. (2020) Suin, M.; Purohit, K.; Rajagopalan, A.; and A. 2020.  Spatially-attentive patch-hierarchical network for adaptive motion deblurring.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3606–3615.
  * Sun et al. (2015) Sun, J.; Cao, W.; Xu, Z.; and Ponce, J. 2015.  Learning a convolutional neural network for non-uniform motion blur removal.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 769–777.
  * Tian et al. (2020) Tian, C.; Fei, L.; Zheng, W.; Xu, Y.; Zuo, W.; and Lin, C.-W. 2020.  Deep learning on image denoising: An overview.  _Neural Networks_ , 131: 251–275.
  * Tran et al. (2021) Tran, P.; Tran, A. T.; Phung, Q.; and Hoai, M. 2021.  Explore image deblurring via encoded blur kernel space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11956–11965.
  * Tu et al. (2022) Tu, Z.; Talebi, H.; Zhang, H.; Yang, F.; Milanfar, P.; Bovik, A.; and Li, Y. 2022\.  Maxim: Multi-axis mlp for image processing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5769–5780.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2022) Wang, Z.; Cun, X.; Bao, J.; Zhou, W.; Liu, J.; and Li, H. 2022.  Uformer: A general u-shaped transformer for image restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 17683–17693.
  * Yang et al. (2019) Yang, W.; Zhang, X.; Tian, Y.; Wang, W.; Xue, J.-H.; and Liao, Q. 2019.  Deep learning for single image super-resolution: A brief review.  _IEEE Transactions on Multimedia_ , 21(12): 3106–3121.
  * Zamir et al. (2022) Zamir, S. W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F. S.; and Yang, M.-H. 2022\.  Restormer: Efficient transformer for high-resolution image restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5728–5739.
  * Zamir et al. (2021) Zamir, S. W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F. S.; Yang, M.-H.; and Shao, L. 2021.  Multi-stage progressive image restoration.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 14821–14831.
  * Zhang et al. (2019) Zhang, H.; Dai, Y.; Li, H.; and Koniusz, P. 2019.  Deep stacked hierarchical multi-patch network for image deblurring.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5978–5986.
  * Zhang et al. (2020) Zhang, K.; Luo, W.; Zhong, Y.; Ma, L.; Stenger, B.; Liu, W.; and Li, H. 2020.  Deblurring by realistic blurring.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2737–2746.