# A Deep Decomposition Network
for Image Processing:
A Case Study for Visible and Infrared Image Fusion

Yu Fu, Xiao-Jun Wu, Josef Kittler

###### Abstract

Image decomposition is a crucial subject in the field of image processing. It
can extract salient features from the source image. We propose a new image
decomposition method based on convolutional neural network. This method can be
applied to many image processing tasks. In this paper, we apply the image
decomposition network to the image fusion task. We input infrared image and
visible light image and decompose them into three high-frequency feature
images and a low-frequency feature image respectively. The two sets of feature
images are fused using a specific fusion strategy to obtain fusion feature
images. Finally, the feature images are reconstructed to obtain the fused
image. Compared with the state-of-the-art fusion methods, this method has
achieved better performance in both subjective and objective evaluation.

###### Index Terms:

image fusion, image decomposition, deep learning, infrared image, visible
image.

##  I Introduction

Image fusion is an important task in image processing. It aims to extract
important features from images of multi-modality signal sources and uses
certain fusion strategies to generate a fused image containing complementary
information of multiple pictures. Our work is one of the common image fusion
tasks, that is, to fuse visible light images and infrared images[1]. The fused
images not only contain the radiation information of the occluded object, but
also retain sufficient texture detail information. At present, many advanced
methods are widely used in production and life, such as security monitoring,
autonomous driving, target tracking, target recognition and other fields.

There are many excellent fusion methods, which can be divided into two
categories: traditional methods and deep learning based methods[2]. Most of
the traditional methods are based on signal processing methods to obtain high-
frequency bands and low-frequency bands of the image and then merge them. With
the development of deep learning, methods based on deep neural networks have
also shown great potential in image fusion, because neural networks can
extract features of source images and perform feature fusion.

Traditional methods can be broadly divided into two categories: one is based
on multi-scale decomposition, and the other is representation learning based
methods. In the multi-scale domain, the image is decomposed into multi-scale
representation feature maps, and then the multi-scale feature representations
are fused through a specific fusion strategy. Finally, the corresponding
inverse transform is used to obtain the fused image. There are many
representative multi-scale decomposition methods, such as pyramid [3],
curvelet[4], contourlet[5],discrete wavelet transform,[6], etc.

In the representation learning domain. The most methods are based on sparse
representation such as sparse representation (SR) and gradient histogram
(HOG)[7], joint sparse representation (JSR)[8], approximate sparse
representation with multi-selection strategy[9], etc.

In the low-rank domain,Li and Wu et al. proposed a low-rank
representation(LRR) based fusion method[10]. The most recent approaches, such
as MDLatLRR[11] are based on image decomposition with Latent LRR. This method
can extract source image features in low-rank domains.

Although the methods based on multi-scale decomposition and representation
learning have achieved good performance. But these methods still have some
problems. These methods are very complicated, and dictionary learning is a
time-consuming operation especially for online training. If the source image
is complex, these methods will not be able to extract the features well.

In order to solve this problem, in recent years, many methods based on deep
learning have been proposed[2] because of the powerful feature extraction
capabilities of neural networks.

In 2017, Liu et al. proposed a method based on convolutional neural network
for multi-focus image fusion[12]. In ICCV2017, Prabhakar et al. proposed
DeepFuse[13] to solve the problem of multi-exposure image fusion. In 2018, Li
and Wu et al. proposed an new infrared and visible light image fusion method
based on denseblock and autoencoder structure[14]. In the next two years, with
the rapid development of deep learning, a large number of excellent methods
emerged. Including IFCNN[15] proposed by Zhang et al., and fusion network
based on GANs (FusionGan)[16] proposed by Ma et al., and the multi-scale
fusion network framework (NestFuse)[17] proposed by Li et al. in 2020. Most of
the methods based on neural networks use the powerful feature extraction
function of neural networks, and then perform fusion at the feature level, and
obtain the final fused image with some specific fusion strategies.

However, the method based on deep network also has some shortcomings: 1. As a
feature extraction tool, neural network cannot explain the meaning of the
extracted features. 2. The network is complex and takes a long time. 3. The
amount and scale of infrared and visible light dataset is small, and many
methods use other data sets for training. This is not necessarily suitable for
extracting infrared and visible light images.

To solve these problems, we propose a novel network that can be used to
decompose images. At the same time, drawing on traditional methods and deep
learning based methods, our proposed network can decompose infrared and
visible light images into high-frequency feature images and low-frequency
feature images to achieve better decomposition effect than traditional
methods. At the same time, we design some fusion rules to fuse the high and
low frequency feature images to obtain the fused feature image. Finally, these
fusion feature images are reconstructed to a fused image. The method we
proposed not only utilizes the powerful feature extraction capabilities of
neural networks, but also realizes the decomposition of image. Compared with
the state-of-the-art methods, our fusion framework has achieved better
performance in both subjective and objective evaluation.

This paper is structured as follows. In Section II, we introduce some related
work. In t Section III, we will introduce our proposed fusion method in
detail. And in Section IV, we illustrate the experimental settings, and we
analyze and compare our experimental results. Finally, in the last section V,
we draw a conclusion of this paper.

##  II RELATED WORKS

Whether it is based on traditional image signal processing methods or deep
learning based methods. They are all very reasonable and excellent methods. We
will introduce some related works that inspired us in this section.

###  II-A Wavelet Decomposition and Laplacian Filter

Wavelet transform has been successfully applied to many image processing
tasks. The most common wavelet transform technique for image fusion is the
Discrete Wavelet Transform (DWT)[18][19].

DWT is a signal processing tool that can decompose signals into high-frequency
information and low-frequency information. Generally speaking, low-frequency
information contains the main characteristics of the signal, and high-
frequency information includes the detailed information of the signal. In the
field of image processing, 2-D DWT is usually used to decompose images. The
wavelet decomposition of the image is given as follows:

$\begin{array}[]{l}M_{LL}(x,y)=\phi(x)\phi(y)\\\ M_{LH}(x,y)=\phi(x)\psi(y)\\\
M_{HL}(x,y)=\psi(x)\phi(y)\\\ M_{HH}(x,y)=\psi(x)\psi(y)\end{array}$ (1)

where $\phi(\cdot)$ is a low-pass filter, and $\psi(\cdot)$ is a high-pass
filter. The input signal $M_{(}x,y)$ is an image with signals in two
directions. Along the $x$ direction and the $y$ direction, high-pass and low-
pass filtering are performed respectively. As shown in Fig.1, we can get a
low-frequency image which is approximate representation and three high-
frequency images which are vertical detail, diagonal detail and horizontal
detail respectively.

Figure 1: Wavelet Decomposition. We perform wavelet decomposition on the image
to get a low-frequency image (b) and three high-frequency images (c)(d)(e) in
three directions.

The Laplacian operator is a simple differential operator with rotation
invariance. The Laplacian transform of a two-dimensional image function is the
isotropic second derivative, defined as:

$\displaystyle\nabla^{2}f(x,y)=\frac{\partial^{2}f(x,y)}{\partial
x^{2}}+\frac{\partial^{2}f(x,y)}{\partial y^{2}}$ (2)

In order to be more suitable for digital image processing, the equation is
approximated as a discrete form:

$\displaystyle\nabla^{2}f(x,y)$ (3)
$\displaystyle\approx\\![f(x\\!+\\!1,y)\\!+\\!f(x\\!-\\!1,y)\\!+\\!f(x,y\\!+\\!1)f(x,y\\!-\\!1)]\\!-\\!4f(x,y)$

The Laplacian operator can also be expressed in the form of a convolution
template, using it as a filtering kernel:

$G_{1}={\left[\begin{array}[]{ccc}0&1&0\\\ 1&-4&1\\\
0&1&0\end{array}\right]},G_{2}={\left[\begin{array}[]{ccc}1&1&1\\\ 1&-8&1\\\
1&1&1\end{array}\right]}$ (4)

$G_{1}$ and $G_{2}$ are the template and the extended template of the discrete
Laplacian operator, and the second differential characteristic of this
template can be used to determine the position of the edge. They are often
used in image edge detection and image sharpening processing, as shown in
Fig.2,.

Figure 2: Laplacian Filter. We use the Laplacian extended template for image
filtering to get its high-frequency image (b), and magnify a local area of the
high-frequency image (c).

We can easily observe that traditional edge filtering is usually just a high-
frequency filtering. While highlighting the edges, they also highlight the
noise.

###  II-B Decomposition-based Fusion Methods

Li and Wu et al. proposed a method[11] to decompose images using low-rank
representation[20].

First, LatLRR[20] can be described as the following optimization problem:

$\begin{array}[]{l}\min_{Z,L,E}\|Z\|_{*}+\|L\|_{*}+\mu\|E\|_{1}\\\
\text{s.t.},X=XZ+LX+E\end{array}$ (5)

Where $\mu$ is a hyper-parameter, and $\|\cdot\|_{*}$ is nuclear norm, and
$\|\cdot\|_{1}$ is $l_{1}$ norm. $X$ is observed data matrix. $Z$ is low-rank
coefficients matrix. $L$ is a projection matrix. $E$ is a sparse noisy matrix.

The author use this method to decompose the image into detail image $I_{d}$
and base image $I_{b}$. We can see from Fig .3 that $I_{d}$ is a high-
frequency image, and $I_{b}$ is a low-frequency image.

Figure 3: The framework of MDLatLRR.

As shown in Fig .3, the low-frequency image $I_{b}$ is continuously decomposed
to obtain several high-frequency image $I_{d1}$, $I_{d2}$ and $I_{d3}$.

Finally, this method decomposes the infrared image and the visible light image
to obtain high-frequency images and low-frequency images. Then we perform a
certain fusion to get the fused image $I_{f}$.

###  II-C Deep Learning-based Fusion Methods

In 2017, Liu et al. proposed a neural network-based method[12]. The authors
divides the picture into many small patches. Then CNN is used to predict
whether each small patch is blurry or clear. The network builds a decision
activation map to indicate which pixels of the original image are clear and
focused. A well-trained network can accomplish multi-focus fusion tasks very
well. However, due to the limitations of network design, this method is only
suitable for multi-focus image fusion.

In order to enable the network to fuse visible light images and infrared
images, Li and Wu et al. proposed a deep neural network (DenseFuse)[14] based
on an autoencoder. First they train a sufficiently powerful encoder and
decoder which can fully extract the features of the original image and
reconstruct the image without losing information as much as possible. Then the
infrared image and the visible light image are inputted into the encoder to
obtain the coding features, and the two sets of features are specifically
fused to obtain the fusion featurs. Finally, the fusion features are inputted
into the decoder to obtain the fused image. These methods use the encoder to
decompose the image into several latent features. Then these features are
fused and reconstructed to obtain a fused image.

In the past few years, Generative Adversarial Networks(GANs) have also been
applied to many fields, including image fusion. In [16] FusionGan first uses
GANs to generate a fused image. The generator inputs infrared and visible
light images and outputs a fused image. In order to improve the quality of the
generated image, the author designed an appropriate loss function. Finally,
the generator can be used to fuse any infrared image and visible light image.

In view of the superiority of these two methods, we propose a multi-layer
image decomposition method based on neural network. And we propose an image
fusion framework for infrared image and visible light image based on this
method.

##  III PROPOSED FUSION METHOD

In this section, the proposed multi-scale decomposition-based fusion network
is introduced in detail. Firstly, the fusion framework is presented in section
III-C. Then, the detail of training phase is described in section III-A. Next,
in section III-B we give the design of the loss function of the network.
Finally, we present different fusion strategy in section 14.

###  III-A Network Structure

In the training phase, we discard the fusion strategy and train the
decomposition network.

Our training goal is to make the decomposition network better decompose the
source image into several high-frequency and one low-frequency images, which
are used for subsequent operations. The structure of the network is shown in
Fig.4, and the detailed network settings are shown in Table I.

Figure 4: The framework of training process.  Figure 5: We use three
convolutions to reduce the channels to one and get a high-frequency image.
Figure 6: We use three convolutions and downsampling twice to reduce the
channels to one to get a low-frequency image. TABLE I: The parameters of the
network

Block | Layer | Channel | Channel | Size | Size | Size | Activation
---|---|---|---|---|---|---|---
(input) | (output) | (kernel) | (input) | (output)
Cin | Conv(Cin-1) | 1 | 16 | 3 | 256 | 256 | LeakyReLU
Conv(Cin-2) | 16 | 32 | 3 | 256 | 256 | LeakyReLU
Conv(Cin-3) | 32 | 64 | 3 | 256 | 256 | LeakyReLU
C1 | Conv(C1) | 64 | 64 | 3 | 256 | 256 | LeakyReLU
C2 | Conv(C2) | 64 | 64 | 3 | 256 | 256 | LeakyReLU
C3 | Conv(C3) | 64 | 64 | 3 | 256 | 256 | LeakyReLU
R1 | Conv(R1) | 64 | 64 | 1 | 256 | 256 | -
R2 | Conv(R2) | 64 | 64 | 1 | 256 | 256 | -
R3 | Conv(R3) | 64 | 64 | 1 | 256 | 256 | -
Detail | Conv(D0) | 64 | 32 | 3 | 256 | 256 | LeakyReLU
Conv(D1) | 32 | 16 | 3 | 256 | 256 | LeakyReLU
Conv(D2) | 16 | 1 | 3 | 256 | 256 | Tanh
C-res | Conv(C-res1) | 64 | 64 | 3 | 256 | 256 | ReLU
Conv(C-res2) | 64 | 64 | 3 | 256 | 256 | ReLU
Conv(C-res3) | 64 | 64 | 3 | 256 | 256 | ReLU
Semantic | Conv(S0) | 64 | 32 | 3 | 256 | 128 | ReLU
Conv(S1) | 32 | 16 | 3 | 128 | 64 | ReLU
Conv(S2) | 16 | 1 | 3 | 64 | 64 | Tanh
Upsample | Upsample | 1 | 1 | - | 64 | 256 | -

In Fig.4 and Table I, $I_{ori}$ is the original input image, and $I_{re}$ is
the reconstructed image. The backbone of the network is four feature
extraction convolutional blocks ($Cin,C1,C2,C3$).

Then the following is the low-frequency feature extraction part, that is, the
${}^{\prime}semantic^{\prime}$ block in figure. The
${}^{\prime}semantic^{\prime}$ block shown in Fig.6 include two down-sampling
convolutional layers ($S0,S1$) with a stride of 2 and a common convolutional
layer ($S3$) which can generate a low-resolution semantic image $I_{s}$. Then
the $I_{s}$ is up-sampled to the same size of the $I_{ori}$ to obtain the low-
frequency image $I_{ups}$.

We copy the features of different depths ($C1,C2,C3$) and and then reshuffle
their channels with convolutional layers ($R1,R2,R3$). After that, we input
them into the ${}^{\prime}detail^{\prime}$ branch of the shared weight to
obtain three high-frequency images $I_{g1}$, $I_{g2}$ and $I_{g3}$. The detail
branch here is shown in detail in Fig.5 and the Table I, which includes three
convolutions (D0, D1, D2), and the number of channels is reduced to 1 to
obtain a high-frequency image.

The reason for adding reshuffle layers($R1,R2,R3$) here is that the detail
block is weight-sharing, the feature maps they extract high-frequency
information should follow the same channel distribution. So we add a $1\times
1$ convolutional layer that does not share weights, and reshuffle and sort the
channels of the features so that the features can adapt to the weight-shared
details block.

Finally, the three high-frequency images ($I_{g1}$, $I_{g2}$, $I_{g3}$) and
one low-frequency image ($I_{ups}$) are added pixel by pixel to obtain the
final reconstructed image $I_{re}$ .

Here we observe that the final reconstructed image is obtained by adding the
high frequency image and the low frequency image. Therefore, the high-
frequency image and the low-frequency image should be a complementary
relationship in the data distribution space. When the network learns to
generate images, the high-frequency image should be the residual data of the
low-frequency image. So we design the residual branch
(${}^{\prime}C-res^{\prime}$ block). We skip-connect the result of $cin$ to
the front of the $semantic$ block, add it to the result of $c3$, and input it
to the following layers. In this way, what $C1,C2$ and $C3$ get are the
residual data between the source image and the semantic image, which is
compulsive and natural. In order to make the skip-connected data more closely
match the deep features of $C3$, we performed three convolutions in
${}^{\prime}C-res^{\prime}$ block to increase the semantics of the skip-
connected features.

As shown in the activation function in the Table I, we consider some
properties of low-frequency images and high-frequency images, we choose
LeakyRelu[21] as the activation function of the convolution of the backbone
network and the high frequency part ($Cin,C1,C2,C3,detail$), and Relu fuction
is used as the activation function of the convolution layers of the residual
branch (${}^{\prime}C-res^{\prime}$) and the $semantic$ block ($S0,S1,S2$).
Because the output of Relu has a certain degree of sparseness, which allows
our low-frequency features to filter out more useless information and retain
more blurred but semantic information. Finally, in order to constrain the
pixel value of the obtained image to a controllable range, we use the $Tanh$
activation function after the last layer of $detail$ and $semantic$ block.

In general, we first perform a convolution block ($cin$) to obtain a set of
feature maps containing various features. Then after three same convolution
operations($c1,c2,c3$), three sets of shallow features are obtained. Then
after two downsampling ($semantic$), deep feature is obtained. We believe that
shallow features contain more low-level information such as texture and
detailed features. We reshuffle the channels and feed these three sets of
shallow features into the high-frequency branch$(detail$) to obtain three
high-frequency images. What is more, we believe that deep feature has more
semantic information and global information, so we convolve and upsample the
deep feature to get our low-frequency images. At the same time, we use the
residual branch ($C-res$) to explicitly establish the residual relationship
between the high-frequency feature and the low-frequency feature. Lastly, we
add these feature images pixel by pixel to get a reconstructed image.

Figure 7: The component of loss function.

###  III-B Loss Fuction

Figure 8: Decomposition and fusion of infrared and visible images.

In the training phase, the loss function(${Loss}_{total}$) of our network
consists of three parts. These losses are the gradient loss(${L}_{detail}$) of
the high-frequency image, the distribution loss (${L}_{semantic}$) of the low-
frequency image and the content reconstruction loss(${L}_{reconstruction}$) of
the reconstructed image. The formula of the loss function is defined as
follows:

$\displaystyle{{Loss}_{total}}={{L}_{detail}}+\alpha{{L}_{semantic}}+\beta{{L}_{reconstruction}}$
(6)

$\alpha$ and $\beta$ are hyper-parameters that balances the three losses.

As shown in Fig. 7, Where ${L}_{detail}$ is to calculate the mean square error
loss between the high-frequency feature map ($I_{g1}$, $I_{g2}$, $I_{g3}$) and
the gradient image of the original image, and then we accumulate these three
losses. The detailed calculation formula of $L_{detail}$ is presented as
follows:

$\displaystyle{{L}_{detail}}={L_{grd\text{-}1}}+{L_{grd\text{-}2}}+{L_{grd\text{-}3}}$
(7) $\displaystyle{{L}_{grd\text{-}i}}$
$\displaystyle=MSE(Gradient({{I}_{ori}}),{{I}_{g\text{-}i}}),i\in\\{{1,2,3}\\}$
$\displaystyle MSE(X,Y)=\frac{1}{N}\sum_{n=1}^{N}(X_{n}-Y_{n})^{2}$

Where $I_{ori}$ is the input source image and $I_{g\text{-}i}$ is the $ith$
high-frequency image. The $MSE(X,Y)$ is the mean square error between $X$ and
$Y$. The gradient image of the original image is obtained by using the
Laplacian gradient operator $Gradient()$. The Laplacian operator performs a
mathematical convolution operation in Equ.4.

In Equ.6, ${L}_{semantic}$ is a data distribution loss. We calculate a strong
supervised loss of the high-frequency image, and calculate a strong supervised
loss of the reconstructed image below. We hope that the low-frequency semantic
block learn to extract deep semantic information, rather than giving it an
answer to let it remember the answer. At the same time, we cannot give a
suitable low-frequency image to the network for reference. The low-frequency
information is definitely not a simply down-sampled image. But if the network
does not have any loss function, it is difficult to get the low-frequency
image we really want. Therefore, we use the down-sampled images as an
approximate data distribution of low frequency images, so that the low-
frequency results generated by our network can be in the ”low frequency
domain” space. The experiment in the next section proves that this loss is
indeed very effective.

$\displaystyle{{L}_{semantic}}=L_{adv}(I_{s},I_{down})$ (8)

where $I_{s}$ is the low-frequency semantic image generated by the network,
$I_{down}$ is the low-frequency blurred image obtained by downsampling the
source image twice, and $L_{adv}$ is the adversarial loss.

$\displaystyle
L_{adv}(I_{s},I_{down})=L_{G}=\frac{1}{N}\sum_{n=1}^{N}\left(D\left(G\left(I_{s}^{n}\right)\right)-1\right)^{2}$
(9) $\displaystyle
L_{D}=\frac{1}{N}\sum_{n=1}^{N}{\left(D\left(I_{down}^{n}\right)-1\right)^{2}}+\frac{1}{N}\sum_{n=1}^{N}{\left(D\left(I_{s}^{n}\right)-0\right)^{2}}$

where $n\in\mathbb{N}_{N}$, $N$ represents the number of images. The loss
function we use here is defined in LSGAN[22].

In Equ.6, $L_{reconstruction}$ is the image content reconstruction loss of the
reconstructed image. The $L_{reconstruction}$ loss consists of two parts, one
is the pixel-level reconstruction loss ${L}_{pix}$, and the other is the
structural similarity loss ${L}_{ssim}$ as follows:

$\displaystyle{{L}_{reconstruction}}={{L}_{pix}}+\gamma{{L}_{ssim}}$ (10)

Where $\gamma$ is a hyper-parameter that balances the two losses. $L_{pixel}$
and $L_{ssim}$ are calculated as follows:

$\displaystyle{{L}_{pixel}}=MSE({{I}_{ori}},{{I}_{re}})$ (11)
$\displaystyle{{L}_{ssim}}=1-SSIM({{I}_{ori}},{{I}_{re}})$ $\displaystyle
SSIM\left(x,y\right)$ $\displaystyle=\
\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})}$

As shown in the Fig. 7, the total loss function $L_{total}$ is given as
follows:

$\displaystyle{{Loss}_{total}}={{L}_{detail}}+\alpha{{L}_{semantic}}+\beta{{L}_{reconstruction}}$
(12)
$\displaystyle={L_{grd\text{-}1}}+{L_{grd\text{-}2}}+{L_{grd\text{-}3}}+\lambda_{1}L_{adv}+\lambda_{2}{{L}_{pix}}+\lambda_{3}{{L}_{ssim}}$

$\lambda_{1},\lambda_{2},\lambda_{3}$ are hyper-parameters and are used to
balance the losses.

###  III-C Image Fusion

In the testing phase, our fusion structure is divided into two parts:
decomposition and fusion, as shown in Fig.9. The decomposition network can
decompose the image into three high-frequency images and one low-frequency
image. The fusion strategy (”FS” in Fig.9) can fuse the corresponding feature
images and reconstruct them to obtain the final image.

Figure 9: The framework of proposed method. ”Decomposition Network” can
decompose the image and ”FS” indicates fusion strategy.

In Fig.9, $I_{ir}$ and $I_{vi}$ represent infrared image and visible light
image, respectively. The two images are fed into the decomposition network to
obtain two sets of feature images. One group of feature images comes from
visible light images including three visible light high-frequency images
($I_{vi\text{-}g1}$, $I_{vi\text{-}g2}$, $I_{vi-g3}$) and one visible light
low-frequency image ($I_{vi\text{-}ups}$). And another group of feature images
comes from infrared images including three infrared high-frequency images
($I_{ir\text{-}g1}$, $I_{ir\text{-}g2}$, $I_{ir\text{-}g3}$) and one infrared
low frequency image ($I_{ir\text{-}ups}$). For the corresponding four groups
of feature images, our fusion strategy contains a variety of fusion methods to
obtain the final fused image $I_{f}$.

In the following subsection, we will introduce the fusion strategy.

###  III-D Fusion strategy

We design a fusion strategy to get a fused image. As shown in the Fig.8, we
first use the decomposition network to decompose the visible light image
$I_{vi}$ and the infrared image $I_{ir}$ to obtain two sets of high and low
frequency feature images. The corresponding high-frequency and low-frequency
feature images (such as $I_{vi\text{-}g1}$ and $I_{ir\text{-}g1}$) are fused
using different specific fusion strategies to obtain fused high-frequency
feature images and low-frequency feature images ($I_{f\text{-}g1}$,
$I_{f\text{-}g2}$, $I_{f\text{-}g3}$, $I_{f\text{-}ups}$). Finally, the fusion
feature image is added pixel by pixel to obtain the fused image $I_{f}$, which
is the same as reconstructing an image in the training phase.

We designed two fusion strategies for high-frequency image fusion, namely,
pixel-wise addition (addition) and the corresponding pixel taking the maximum
value (max). In addition, we also designed two fusion methods for low-
frequency images, which are adding and averaging pixel by pixel (avg), and the
corresponding pixel takes the maximum value (max), as shown in Fig.10.

Figure 10: High and low frequency image fusion strategy. Here a and b are any
pair of points from feature images, and c is the corresponding fused pixel.

The formulas of high frequency fusion feature $I_{f\text{-}g}$ and low
frequency fusion feature $I_{f\text{-}ups}$ are described as follows:

$\displaystyle
I_{f\text{-}gj}^{n}=max(I_{vi\text{-}gi}^{n},I_{ir\text{-}gi}^{n})\ ,$
$\displaystyle(Max)$ (13) $\displaystyle or$ $\displaystyle
I_{f\text{-}gi}^{n}=I_{vi\text{-}gi}^{n}+I_{ir\text{-}gj}^{n}$
$\displaystyle(Add)$ $\displaystyle
I_{f\text{-}ups}^{n}=\frac{(I_{vi\text{-}ups}^{n},I_{ir\text{-}ups}^{n})}{2}\
,$ $\displaystyle(Avg)$ $\displaystyle or$ $\displaystyle
I_{f\text{-}ups}^{n}=max(I_{vi\text{-}ups}^{n},I_{ir\text{-}ups}^{n})$
$\displaystyle(Max)$ $\displaystyle i\in\\{{1,2,3}\\},n\in\mathbb{N}$

Where $i$ represents three high-frequency images, and $\mathbb{N}$ represents
all pixels in the image. $I_{vi\text{-}gi}^{n}$ and $I_{ir\text{-}gi}^{n}$ are
any pixel in the corresponding three groups of high-frequency images, and
$I_{vi\text{-}ups}^{n}$ and $I_{ir\text{-}ups}^{n}$ are any pixel in the low-
frequency image. We calculate and fuse the corresponding pixels to get the
pixels of the fused high frequency image $I_{f\text{-}gi}^{n}$ and low
frequency image $I_{f\text{-}ups}^{n}$. Finally, the three fusion features are
added to obtain the final fused image $I_{f}$ as follows:

${{I}_{f}}={{I}_{f-g1}}+{{I}_{f-g2}}+{{I}_{f-g3}}+{{I}_{f-ups}}$ (14)

##  IV EXPERIMENTS AND ANALYSIS

###  IV-A Training and Testing Details

For the selection of hyper-parameters, we make the values of losses as close
to the same order of magnitude as possible. So, in formula 12, we set
$\lambda_{1}$ = 0.1, $\lambda_{2}$ = 100, $\lambda_{3}$ = 10 by cross
validation.

Our goal is to train a powerful decomposition network that can decompose
images into high-frequency and low-frequency images well. In this way, our
input images in the training phase are not limited to infrared images and
visible light images. We can also use MS-COCO[23] and Imagenet[24] or other
images to achieve this goal. In our experiment, we use MS-COCO as the training
set to train our decomposition network. We select about 80,000 images as input
images. These images are converted to gray scale images which are then resized
to 256$\times$256.

We select twelve pairs of infrared and visible light images from the TNO [25]
as our test images. The reason why the TNO dataset is not used as training
data is that the TNO dataset has few pictures and is suitable for testing. At
the same time, we select fifty pairs of infrared and visible light images from
the RoadScene dataset[26] for testing.

We input batchsize of 64 images to the network every iteration. And, we select
Adam[27] iterator and adaptive learning rate decay method[28] as the learning
rate scheduler. We set the initial learning rate to 1e-3, the attenuation
factor to 0.5, the maximum patience to 5 iterations, and the minimum learning
rate threshold to 1e-8. We set the maximum number of epoches to 1000.

In the test phase, because our network is a fully convolutional network, we
input infrared images and visible light images without preprocessing
operations.

The experiment is conducted on the two NVIDIA TITAN Xp GPUs and 128GB of CPU
memory. We decompose 1000 images with 256$\times$256 resolution one by one and
calculate the average calculation time. It takes about 2ms to decompose each
image.

###  IV-B the role of the adversarial loss

As shown in Fig 11, if we do not give constraints on the low-frequency image
$I_{s}$, it is difficult for the network to learn smartly to get a semantic
low-frequency image we want. Without the distribution loss function, the high-
frequency images learned by the network have too much semantic information,
such as the distribution of colors-this is not high-frequency information. And
low-frequency images loses a lot of semantic information.

In order to allow the $semantic$ block to learn the real low-frequency
information we want, we give it a hint that is the weak supervision loss. As
in Equ. 9, we regard the down-sampled image $I_{down}$ as an approximate
solution of the low-frequency image, so that the low-frequency image $I_{s}$
generated by the network follows the distribution of the low-frequency images.

Figure 11: The effect of adversarial loss. a) is the original image, b) and c)
is the high-frequency image and low-frequency image without adversarial loss,
and d) and e) is the result of using adversarial loss.

###  IV-C the details of the decomposed images

Although the loss function of our high-frequency image is calculated with
gradient map which is calculated by Laplacian operator. But the result of our
high-frequency image is quiet different from the Laplacian gradient high-
frequency image.

As shown in Fig 12, we list the high-frequency images decomposed by the
proposed decomposition network and the Laplacian operator. It can be seen that
the Laplacian gradient images only extract part of the high-frequency
information, and the image has a lot of noises.

Figure 12: High-frequency images decomposed by the proposed decomposition
network and the Laplacian operator.

The high-frequency image decomposed by our decomposition network not only
retains almost all high-frequency information on the basis of the Laplacian
gradient image, but also completely extracts the contour and detail
information of the object. In addition, our high-frequency images have a
certain degree of semantic recognition, and can clearly express the semantic
features according to the outline of the objects.

Figure 13: Experiment on street images.

###  IV-D Comparison with State of The Art Methods

We select ten classic and the state of the art fusion methods to compare the
fusion effect of our proposed method, including Curvelet Transform (CVT)[29],
dualtree complex wavelet transform (DTCWT)[30], Multi-resolution Singular
Value Decomposition (MSVD)[31], DenseFuse[14], the GAN-based fusion network
(FusionGAN)[16], a general end-to-end fusion network(IFCNN)[15], MDLatLRR[11],
NestFuse[17], FusionDN[26] and U2Fusion[32]. We use the public codes of these
methods and the parameters shown in the paper to obtain fused images.

Because there is currently no clear specific evaluation indicators to measure
the quality of the fused image, we will comprehensively compare it according
to the subjective evaluation and the objective evaluation respectively.

####  IV-D1 subjective evaluation

In different fields, for different tasks, everyone has his/her own criteria
for judging. We consider the subjective feelings of the picture, such as
lightness, fidelity, noise, and clarity etc.

Figure 14: Experiment on road images.

In Fig. 13 and Fig. 14, our method is compared with other methods. It can be
clearly seen that our fused image not only perfectly retains the radiation
information of the infrared image, but also fully retains the detailed texture
information of the visible light image. More importantly, our image does not
have a lot of noises. We marked some salient areas with red boxes. For
example, in Fig. 13, the canopy of the shop has less noises. In Fig. 14, the
outline of the person in the distance is clearly visible..

TABLE II: OBJECTIVE EVALUATION OF CLASSIC AND LATEST FUSION ALGORITHMS ON TNO
DATASET

Methods | EI | SF | EN | SCD | FMI_w | FMI_dct | MI | SD | DF | AG | QG
---|---|---|---|---|---|---|---|---|---|---|---
CVT | 42.9631 | 11.1129 | 6.4989 | 1.5812 | 0.4240 | 0.3945 | 12.9979 | 27.4613 | 5.4530 | 4.2802 | 0.4623
DTCWT | 42.4889 | 11.1296 | 6.4791 | 1.5829 | 0.4419 | 0.3936 | 12.9583 | 27.3099 | 5.4229 | 4.2370 | 0.5024
MSVD | 27.6098 | 8.5538 | 6.2807 | 1.5857 | 0.2828 | 0.2470 | 12.5613 | 24.0288 | 4.2283 | 2.8773 | 0.3375
DenseFuse | 36.4838 | 9.3238 | 6.8526 | 1.5329 | 0.4389 | 0.3897 | 13.7053 | 38.0412 | 4.6176 | 3.6299 | 0.4569
FusionGan | 32.5997 | 8.0476 | 6.5409 | 0.6876 | 0.4083 | 0.4142 | 13.0817 | 29.1495 | 4.2727 | 3.2803 | 0.2784
IFCNN | 44.9725 | 11.8590 | 6.6454 | 1.6126 | 0.4052 | 0.3739 | 13.2909 | 33.0086 | 5.9808 | 4.5521 | 0.4864
MDLatLRR | 28.0985 | 7.3383 | 6.3016 | 1.6043 | 0.4296 | 0.4080 | 12.6032 | 24.7217 | 3.5486 | 2.7938 | 0.4127
NestFuse | 38.4401 | 9.7098 | 6.8856 | 1.5839 | 0.4504 | 0.3694 | 13.7713 | 38.3311 | 4.9099 | 3.8376 | 0.4895
FusionDN | 61.3491 | 14.2256 | 7.4073 | 1.6148 | 0.3651 | 0.3159 | 13.6147 | 48.5659 | 7.4565 | 5.9832 | 0.3785
U2Fusion | 48.4915 | 11.0368 | 6.7227 | 1.5946 | 0.3594 | 0.3381 | 13.4453 | 31.3794 | 5.8343 | 4.7392 | 0.4039
| max + avg | 30.1185 | 8.0322 | 6.3621 | 0.7133 | 0.1282 | 0.0984 | 12.7242 | 25.2654 | 3.9611 | 3.0370 | 0.1032
| max + max | 33.5991 | 8.4408 | 6.6711 | 0.6959 | 0.1299 | 0.0985 | 13.3422 | 38.5077 | 4.1961 | 3.3191 | 0.1023
| add + avg | 46.0222 | 12.0192 | 6.5235 | 0.6629 | 0.1278 | 0.0984 | 13.0470 | 27.4135 | 6.1148 | 4.6537 | 0.1017
Ours | add + max | 48.5475 | 12.3100 | 6.8973 | 0.6748 | 0.1293 | 0.0984 | 13.7945 | 39.9003 | 6.2682 | 4.8526 | 0.1019

TABLE III: OBJECTIVE EVALUATION OF CLASSIC AND LATEST FUSION ALGORITHMS ON
RoadScene DATASET

Methods | EI | SF | EN | SCD | $FMI_{w}$ | $FMI_{dct}$ | MI | SD | DF | AG | QG
---|---|---|---|---|---|---|---|---|---|---|---
CVT | 59.7642 | 14.7379 | 7.0159 | 1.3418 | 0.4138 | 0.3631 | 14.0319 | 36.0884 | 6.9618 | 5.7442 | 0.4499
DTCWT | 57.3431 | 14.7318 | 6.9211 | 1.3329 | 0.3458 | 0.2383 | 13.8421 | 34.7264 | 6.7810 | 5.5228 | 0.4402
MSVD | 36.0475 | 11.3182 | 6.6960 | 1.3458 | 0.2659 | 0.2195 | 13.3919 | 30.9643 | 5.0926 | 3.6171 | 0.3600
DenseFuse | 34.0135 | 8.5541 | 6.6740 | 1.3491 | 0.4173 | 0.3857 | 13.3480 | 30.6655 | 3.9885 | 3.2740 | 0.3916
FusionGan | 35.4048 | 8.6400 | 7.1753 | 0.8671 | 0.3410 | 0.3609 | 14.3507 | 42.3040 | 3.9243 | 3.3469 | 0.2591
IFCNN | 57.6653 | 15.0677 | 6.9730 | 1.3801 | 0.4032 | 0.3456 | 13.9460 | 35.8183 | 7.0401 | 5.6242 | 0.5100
MDLatLRR | 36.9468 | 9.3638 | 6.7171 | 1.3636 | 0.4241 | 0.3875 | 13.4342 | 31.3505 | 4.3216 | 3.5530 | 0.4483
NestFuse | 53.9286 | 14.2820 | 7.3598 | 1.2597 | 0.4342 | 0.3484 | 14.7196 | 48.9920 | 6.2840 | 5.1834 | 0.4758
FusionDN | 63.1690 | 16.7138 | 7.5323 | 1.1882 | 0.3621 | 0.3009 | 15.0646 | 55.0559 | 7.7925 | 6.4950 | 0.4631
U2Fusion | 66.2529 | 15.8242 | 7.1969 | 1.3551 | 0.3717 | 0.3199 | 14.3938 | 42.9368 | 7.5930 | 6.3133 | 0.5112
| max + avg | 39.4996 | 10.7670 | 6.7575 | 1.3394 | 0.3151 | 0.2311 | 13.5150 | 31.6977 | 4.8180 | 3.8643 | 0.3591
| max + max | 39.7592 | 10.6215 | 6.8186 | 1.3223 | 0.3171 | 0.2217 | 13.6371 | 39.6907 | 4.7275 | 3.8575 | 0.3505
| add + avg | 64.5832 | 16.5082 | 6.9535 | 1.4087 | 0.4295 | 0.3883 | 13.9071 | 35.7447 | 7.7606 | 6.2775 | 0.4969
Ours | add + max | 63.4740 | 16.1703 | 6.9666 | 1.3774 | 0.4076 | 0.3640 | 13.9332 | 43.2688 | 7.5247 | 6.1431 | 0.4478

####  IV-D2 objective evaluation

Subjective feelings have great personal factors, and it is not enough for
evaluation to rely solely on subjective evaluation. We select fifteen
objective evaluation indicators from the popular objective indicators for
comprehensive evaluation. They are: Edge Intensity(EI)[33], SF[34], Entropy
(EN)[35], Sum of Correlation Coefficients (SCD)[36], Fast Mutual Information
($FMI_{w}$ and $FMI_{dct}$)[37] ,Mutual Information (MI)[38], Standard
Deviation of Image (SD), Definition (DF)[39], Average gradient (AG)[40] and
QG[33] respectively.

The objective evaluation indicators here are divided into two categories. One
is to evaluate the fused image, such as calculating the edge(EI), the number
of mutations in the image(SF), average gradient(AG), entropy (EN), clarity(DF)
and contrast of the image(SD). The other is to evaluate the fused image with
the source image. There is another category that evaluates the fused image and
the source image, such as the mutual information (MI, $FMI_{w}$ and
$FMI_{dct}$) and some complex calculation methods(QG).

We compare the proposed method with ten other excellent methods, and the
results of the average values for all fused images shown in Table II and Table
III respectively. The best value in the quality table is made bold in red and
bold, and the second best value is given in _bold and italic_.

It can be seen from Table II and Table III that our proposed method obtains
eleven best values and four second best values. On the TNO dataset, our method
($add+max$) obtains one best result and six second best results. On the
RoadScene dataset, our method ($add+avg$) obtained two best results and four
second best results. Comparing with other operator in fusion strategy, the
addition operator of high frequency information can achieve better results in
several indicators. Although our method does not get the best results in every
result, it was able to get second best results in many indicators.

##  V CONCLUSIONS

In this paper, we propose a novel multi-network for image decomposition. We
also develop a decomposition network fusion framework to fuse infrared images
and visible light images. Firstly, with the help of decomposition networks,
the infrared image and the visible light image are decomposed into multiple
high-frequency feature images and a low-frequency feature image, respectively.
Secondly, the corresponding feature image is fused with a specific fusion
strategy to obtain the fusion feature images. Finally, the fusion feature
images are added pixel by pixel to obtain the fused image. This kind of image
decomposition network is universal, and any number of images can be quickly
and effectively decomposed by neural network. At the same time, using the
power of GPUS, neural networks can easily use GPU for matrix calculation
acceleration. The speed of image decomposition can also be very fast. We have
performed a subjective and objective evaluation of the proposed method, and
the experimental results show that it has reached the state of the art.
Although the network structure is simple, it proves the feasibility of the
neural network to decompose the image. We have a conjecture that CNN uses the
semantics of the image to filter the noise while preserving the edges, and
obtain a very good high-frequency information image. We will continue to study
image decomposition based on deep learning, including simplifying some
originally complex image decomposition calculations such as wavelet
transformation, low-rank decomposition, etc., or designing more reasonable
network structures for other image processing applications. We think that the
network we propose can be used for different image processing tasks, including
multi-focus fusion, medical image fusion, multi-exposure fusion, and some
basic computer vision tasks such as detection, recognition, and
classification. We will then experiment and test this method in other image
tasks.

## References

  * [1] J. Ma, Y. Ma, and C. Li, “Infrared and visible image fusion methods and applications: A survey,” _Information Fusion_ , vol. 45, pp. 153–178, 2019\.
  * [2] Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward, and X. Wang, “Deep learning for pixel-level image fusion: Recent advances and future prospects,” _Information Fusion_ , vol. 42, pp. 158–173, 2018.
  * [3] T. Mertens, J. Kautz, and F. Van Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,” _Computer Graphics Forum_ , vol. 28, no. 1, pp. 161–171, 2009.
  * [4] Z. Zhang and R. S. Blum, “A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application,” _Proceedings of the IEEE_ , vol. 87, no. 8, pp. 1315–1326, 1999.
  * [5] K. P. Upla, M. V. Joshi, and P. P. Gajjar, “An edge preserving multiresolution fusion: Use of contourlet transform and mrf prior,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 53, no. 6, pp. 3210–3220, 2014.
  * [6] A. B. Hamza, Y. He, H. Krim, and A. S. Willsky, “A multiscale approach to pixel-level image fusion,” _Computer-Aided Engineering_ , vol. 12, no. 2, pp. 135–146, 2005.
  * [7] J.-j. Zong and T.-s. Qiu, “Medical image fusion based on sparse representation of classified image patches,” _Biomedical Signal Processing and Control_ , vol. 34, pp. 195–205, 2017.
  * [8] Q. Zhang, Y. Fu, H. Li, and J. Zou, “Dictionary learning method for joint sparse representation-based image fusion,” _Optical Engineering_ , vol. 52, no. 5, p. 057006, 2013.
  * [9] Y. Bin, Y. Chao, and H. Guoyu, “Efficient image fusion with approximate sparse representation,” _International Journal of Wavelets, Multiresolution and Information Processing_ , vol. 14, no. 04, p. 1650024, 2016.
  * [10] H. Li and X.-J. Wu, “Multi-focus image fusion using dictionary learning and low-rank representation,” in _International Conference on Image and Graphics_. Springer, 2017, pp. 675–686.
  * [11] H. Li, X.-J. Wu, and J. Kittler, “Mdlatlrr: A novel decomposition method for infrared and visible image fusion,” _IEEE Transactions on Image Processing_ , 2020.
  * [12] Y. Liu, X. Chen, H. Peng, and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” _Information Fusion_ , vol. 36, pp. 191–207, 2017.
  * [13] K. R. Prabhakar, V. S. Srikar, and R. V. Babu, “Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs.” in _ICCV_ , 2017, pp. 4724–4732.
  * [14] H. Li and X.-J. Wu, “Densefuse: A fusion approach to infrared and visible images,” _IEEE Transactions on Image Processing_ , vol. 28, no. 5, pp. 2614–2623, 2018.
  * [15] Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao, and L. Zhang, “Ifcnn: A general image fusion framework based on convolutional neural network,” _Information Fusion_ , vol. 54, pp. 99–118, 2020.
  * [16] J. Ma, W. Yu, P. Liang, C. Li, and J. Jiang, “Fusiongan: A generative adversarial network for infrared and visible image fusion,” _Information Fusion_ , vol. 48, pp. 11–26, 2019.
  * [17] H. Li, X.-J. Wu, and T. Durrani, “Nestfuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models,” _IEEE Transactions on Instrumentation and Measurement_ , 2020.
  * [18] H. Li, B. S. Manjunath, and S. K. Mitra, “Multisensor image fusion using the wavelet transform,” _Graphical Models and Image Processing_ , vol. 57, no. 3, pp. 235–245, 1995.
  * [19] L. J. Chipman, T. M. Orr, and L. N. Graham, “Wavelets and image fusion,” vol. 3, p. 3248, 1995.
  * [20] G. Liu and S. Yan, “Latent low-rank representation for subspace segmentation and feature extraction,” pp. 1615–1622, 2011.
  * [21] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. icml_ , vol. 30, no. 1, 2013, p. 3.
  * [22] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley, “Least squares generative adversarial networks,” pp. 2813–2821, 2017.
  * [23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [24] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [25] A. Toet _et al._ , “Tno image fusion dataset,” _Figshare. data_ , 2014\.
  * [26] H. Xu, J. Ma, Z. Le, J. Jiang, and X. Guo, “Fusiondn: A unified densely connected network for image fusion.” in _AAAI_ , 2020, pp. 12 484–12 491.
  * [27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv: Learning_ , 2014.
  * [28] M. D. Zeiler, “Adadelta: An adaptive learning rate method,” _arXiv: Learning_ , 2012.
  * [29] F. Nencini, A. Garzelli, S. Baronti, and L. Alparone, “Remote sensing image fusion using the curvelet transform,” _Information fusion_ , vol. 8, no. 2, pp. 143–156, 2007.
  * [30] J. J. Lewis, R. J. Callaghan, S. G. Nikolov, D. R. Bull, and N. Canagarajah, “Pixel-and region-based image fusion with complex wavelets,” _Information fusion_ , vol. 8, no. 2, pp. 119–130, 2007.
  * [31] V. Naidu, “Image fusion technique using multi-resolution singular value decomposition,” _Defence Science Journal_ , vol. 61, no. 5, p. 479, 2011\.
  * [32] H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, “U2fusion: A unified unsupervised image fusion network,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [33] C. S. Xydeas and V. S. Petrovic, “Objective pixel-level image fusion performance measure,” in _Sensor Fusion: Architectures, Algorithms, and Applications IV_ , vol. 4051. International Society for Optics and Photonics, 2000, pp. 89–98.
  * [34] A. M. Eskicioglu and P. S. Fisher, “Image quality measures and their performance,” _IEEE Transactions on communications_ , vol. 43, no. 12, pp. 2959–2965, 1995.
  * [35] J. W. Roberts, J. A. van Aardt, and F. B. Ahmed, “Assessment of image fusion procedures using entropy, image quality, and multispectral classification,” _Journal of Applied Remote Sensing_ , vol. 2, no. 1, p. 023522, 2008.
  * [36] V. Aslantas and E. Bendes, “A new image quality metric for image fusion: the sum of the correlations of differences,” _Aeu-international Journal of electronics and communications_ , vol. 69, no. 12, pp. 1890–1896, 2015.
  * [37] M. Haghighat and M. A. Razian, “Fast-fmi: non-reference image fusion metric,” in _2014 IEEE 8th International Conference on Application of Information and Communication Technologies (AICT)_. IEEE, 2014, pp. 1–3.
  * [38] H. Peng, F. Long, and C. Ding, “Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 27, no. 8, pp. 1226–1238, 2005.
  * [39] X. Desheng, “Research of measurement for digital image definition,” _Journal of Image and Graphics_ , 2004.
  * [40] G. Cui, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition,” _Optics Communications_ , vol. 341, pp. 199–209, 2015.

|  Yu Fu received the B.M. degree in professional engineering management from
North China Institute Of Science And Technology, China. He is currently a
Master student in the Jiangsu Provincial Engineerinig Laboratory of Pattern
Recognition and Computational Intelligence, Jiangnan University. His research
interests include image fusion, machine learning and deep learning.
---|---
|  Xiao-Jun Wu received the B.Sc. degree in mathematics from Nanjing Normal
University, Nanjing, China, in 1991, and the M.S. and Ph.D. degrees in pattern
recognition and intelligent system from the Nanjing University of Science and
Technology, Nanjing, in 1996 and 2002, respectively From 1996 to 2006, he
taught at the School of Electronics and Information, Jiangsu University of
Science and Technology, where he was promoted to a Professor. He was a Fellow
of the International Institute for Software Technology, United Nations
University, from 1999 to 2000. He was a Visiting Researcher with the Centre
for Vision, Speech, and Signal Processing (CVSSP), University of Surrey, U.K.,
from 2003 to 2004. Since 2006, he has been with the School of Information
Engineering, Jiangnan University, where he is currently a Professor of pattern
recognition and computational intelligence. His current research interests
include pattern recognition, computer vision, and computational intelligence.
He has published over 300 articles in his fields of research. He was a
recipient of the Most Outstanding Postgraduate Award from the Nanjing
University of Science and Technology.
---|---
|  Josef Kittler (Life Member, IEEE) received the B.A., Ph.D., and D.Sc.
degrees from the University of Cambridge, in 1971, 1974, and 1991,
respectively. He is currently a Distinguished Professor of machine
intelligence at Centre for Vision, Speech and Signal Processing, University of
Surrey, Guildford, U.K. He conducts research on biometrics, video and image
database retrieval, medical image analysis, and cognitive vision. He published
the textbook Pattern Recognition: A Statistical Approach and over 700
scientific papers. His publications have been cited over 60,000 times (Google
Scholar). He is a Series Editor of Lecture Notes in Computer Science
(Springer). He currently serves on the Editorial Boards of Pattern Recognition
Letters, International Journal of Pattern Recognition and Artificial
Intelligence, and Pattern Analysis and Applications. He has also served as a
member of the Editorial Board for the IEEE TRANSACTIONS ON PATTERN ANALYS IS
AND MACHINE INTELLIGENCE from 1982 to 1985. He served on the Governing Board
of the International Association for Pattern Recognition (IAPR), as one of the
two British representatives, from 1982 to 2005, and the President of the IAPR
from 1994 to 1996.
---|---