# Pixel-Aware Stable Diffusion for Realistic Image Super-resolution
and Personalized Stylization

Tao Yang1, Peiran Ren1, Xuansong Xie1, Lei Zhang2<EMAIL_ADDRESS>
###### Abstract

Realistic image super-resolution (Real-ISR) aims to reproduce perceptually
realistic image details from a low-quality input. The commonly used
adversarial training based Real-ISR methods often introduce unnatural visual
artifacts and fail to generate realistic textures for natural scene images.
The recently developed generative stable diffusion models provide a potential
solution to Real-ISR with pre-learned strong image priors. However, the
existing methods along this line either fail to keep faithful pixel-wise image
structures or resort to extra skipped connections to reproduce details, which
requires additional training in image space and limits their extension to
other related tasks in latent space such as image stylization. In this work,
we propose a pixel-aware stable diffusion (PASD) network to achieve robust
Real-ISR as well as personalized stylization. In specific, a pixel-aware cross
attention module is introduced to enable diffusion models perceiving image
local structures in pixel-wise level, while a degradation removal module is
used to extract degradation insensitive features to guide the diffusion
process together with image high level information. By simply replacing the
base diffusion model with a personalized one, our method can generate diverse
stylized images without the need to collect pairwise training data. PASD can
be easily integrated into existing diffusion models such as Stable Diffusion.
Experiments on Real-ISR and personalized stylization demonstrate the
effectiveness of our proposed approach. The source code and models can be
found at https://github.com/yangxy/PASD.

## Introduction

Images often suffer from a mixture of complex degradations, such as low
resolution, blur, noise, etc., in the acquisition process. While image
restoration methods (Yang et al. 2008) have achieved significant progress,
especially in the era of deep learning (Dong et al. 2014; Lim et al. 2017),
they still tend to generate over-smoothed details, partially due to the pursue
of image fidelity in the methodology design. By relaxing the constraint on
image fidelity, realistic image super-resolution (Real-ISR) aims to reproduce
perceptually realistic image details from the degraded observation. The
generative adversarial networks (GANs) (Goodfellow et al. 2014) and the
adversarial training strategy have been widely used for Real-ISR (Ledig et al.
2017; Wang et al. 2018b) and achieved promising results. However, GAN-based
Real-ISR methods tend to generate unpleasant visual artifacts. Some methods
(Zhou et al. 2022) learn discrete codebooks to reduce the uncertainty in GAN-
based face restoration, and some methods (Chen et al. 2022) restore images by
matching distorted features to their distortion-free counterparts.
Nonetheless, the above methods are limited in reproducing rich and realistic
image details.

Recently, denoising diffusion probabilistic models (DDPMs) have shown
outstanding performance in tasks of image generation (Ho, Jain, and Abbeel
2020), image-to-image translation (Saharia et al. 2022a), etc. DDPM is a
strong alternative to GAN in many downstream tasks (Dhariwal and Nichol 2021;
Rombach et al. 2022) due to its powerful capability in approximating diverse
and complicated distributions. Denoising diffusion implicit model (DDIM)
(Song, Meng, and Ermon 2021) was proposed to accelerate the sampling speed of
DDPM. The DDPM/DDIM based pre-trained text-to-image (T2I) and text-to-video
(T2V) models (Rombach et al. 2021; Ramesh et al. 2022; Saharia et al. 2022b;
Ho et al. 2022) have been popularly used in numerous downstream tasks,
including personalized image generation (Ruiz et al. 2023; Kumari et al.
2023), image editing (Hertz et al. 2022; Brooks, Holynski, and Efros 2023;
Kawar et al. 2023), image inpainting (Yang et al. 2022a) and conditional image
synthesis (Zhang and Agrawala 2023). They have also been adopted to solve
image restoration tasks. A denoising diffusion restoration model (DDRM) is
proposed in (Kawar et al. 2022) to solve inverse problem by taking advantage
of a pre-trained denoising diffusion generative model. However, a liner image
degradation model is assumed to be known in DDRM, limiting its application to
more practical scenarios such as Real-ISR.

Considering that the pre-trained T2I models such as Stable Diffusion (SD)
(Rombach et al. 2021) can generate high-quality natural images, Zhang and
Agrawala (Zhang and Agrawala 2023) proposed ControlNet, which enables
conditional inputs like edge maps, segmentation maps, etc., and demonstrated
that the generative diffusion priors are also powerful in conditional image
synthesis. Unfortunately, ControlNet is not suitable for pixel-wise
conditional control (see Fig. 1 for an example). Qin et al. (Qin et al. 2023)
extended ControlNet by introducing UniControl to enable more diverse visual
conditions. Liu et al. (Liu et al. 2023) and Wang et al. (Wang et al. 2023)
demonstrated that pre-trained SD priors can be employed for image colorization
and Real-ISR, respectively. However, they resorted to a skipped connection to
pass pixel-level details for image restoration, requiring extra training in
image space and limiting the model capability to tasks performed in latent
space such as image stylization.

In this work, we investigate the problem of Real-ISR with pre-trained T2I
models such as SD, targeting at reconstructing photo-realistic structures and
textures. Our idea is to introduce pixel-aware conditional control into the
diffusion process so that robust and perceptually realistic Real-ISR results
can be achieved. To this end, we present a pixel-aware cross attention (PACA)
module to perceive pixel-level information without using any skipped
connections. A degradation removal module is employed to reduce the impact of
unknown image degradations, alleviating the burden of diffusion module to
handle real-world low-quality images. We also demonstrate that the high-level
classification/detection/captioning information extracted from the input image
can further boost the Real-ISR performance. Last but not least, the proposed
pixel-aware stable diffusion (PASD) network can perform personalized
stylization tasks by simply shifting the base model to a personalized one.

Figure 1: An input LQ image (left) and the Real-ISR output (right) by
ControlNet. One can see clearly the content inconsistency between them. Figure
2: Architecture of the proposed pixel-aware stable diffusion (PASD) network.

## Related Work

Realistic Image Super-Resolution. Though deep learning based image super-
resolution (Dong et al. 2014; Lim et al. 2017) has achieved significant
progress, they still suffer from over-smoothed details due to the high illness
of the task by minimizing the fidelity objectives (e.g., PSNR, SSIM).
Realistic image super-resolution (Real-ISR) aims to reproduce perceptually
photo-realistic image details by optimizing both fidelity and perception
objectives. The GAN (Goodfellow et al. 2014) network and its adversarial
training strategies are widely in Real-ISR (Ledig et al. 2017; Wang et al.
2018b). Basically, a generator network is used to reconstruct the desired
high-quality (HQ) image from the low-quality (LQ) input, while a discriminator
network is used to judge whether the HQ output is perceptually realistic.
However, adversarial training is unstable, and the GAN-based Real-ISR methods
often bring unnatural visual artifacts. Liang et al. (Liang, Zeng, and Zhang
2022) proposed a locally discriminative learning approach to suppress the GAN-
generated artifacts, yet it is difficult to introduce more details. Recently,
inspired by the success of generative priors in face restoration tasks (Yang
et al. 2021; Wang et al. 2021a), some works exploit the priors extracted from
VQGAN (Esser, Rombach, and Ommer 2021), diffusion model (Ho, Jain, and Abbeel
2020) and pre-trained T2I model (Rombach et al. 2021) to solve the Real-ISR
problems and led to interesting results (Chen et al. 2022; Kawar et al. 2022;
Wang et al. 2023).

In the early study, bicubic downsampling or some simple degradations (Dong et
al. 2014; Lai et al. 2017; Gu et al. 2019) are used to simulate the LQ-HQ
training pairs. Zhang et al. (Zhang et al. 2021) and Wang et al. (Wang et al.
2021b) later modeled complex degradations by shuffling degradation types and
using a high-order process, respectively. Cai et al. (Cai et al. 2019)
collected a real-world dataset with paired LQ-HQ images by zooming camera
lens. Lugmayr et al. (Lugmayr, Danelljan, and Timofte 2019) learned a
distribution mapping network with unpaired data.

Personalized Stylization. Inspired by the powerful learning capacity of deep
neural networks, Gatys et al. (Gatys, Ecker, and Bethge 2015) presented an
optimization based method to transfer the style of a given artwork to a
content image. This work was extended and developed by many following
researchers (Johnson, Alahi, and Li 2016; Li et al. 2017; Zhang et al. 2023).
However, these methods require an extra image as style input. This problem can
be alleviated by resorting to an image-to-image framework (Zhu et al. 2017;
Chen, Lai, and Liu 2018; Chen, Liu, and Chen 2020). Due to the lack of
pairwise training data, some works (Men et al. 2022; Yang et al. 2022b) focus
on portrait stylization with the help of StyleGAN (Karras, Laine, and Aila
2019). With the rapid development of SD models, some works (Brooks, Holynski,
and Efros 2023; Zhang and Agrawala 2023) generate stylized images by using
proper instruction prompts, achieving impressive results. However, these
methods fail to maintain pixel-wise image structures in the stylization
process. In addition, these methods lack the ability to mimic the appearance
of subjects in a given reference set. To meet the specific needs of different
users, Ruiz et al. (Ruiz et al. 2023) and Kumari et al. (Kumari et al. 2023)
proposed personalized stylization approaches for T2I diffusion models.

Diffusion Probabilistic Models. The seminal work of DDPM (Ho, Jain, and Abbeel
2020) demonstrates promising capability in generating high quality natural
images. Considering that DDPMs require hundreds of sampling steps in the
denoising process, Song et al. (Song, Meng, and Ermon 2021) proposed DDIM to
accelerate the sampling speed. Following works extend DDPM/DDIM by adapting
high-order solvers (Lu et al. 2022) and distillations (Meng et al. 2023).
Rombach et al. (Rombach et al. 2022) extended DDPM to latent space and
demonstrated impressive results with less computational resources. This work
sparks the prosperity of large pre-trained T2I and T2V diffusion models such
as SD (Rombach et al. 2021), Imagen (Ho et al. 2022). It has been demonstrated
that T2I diffusion priors are more powerful than GAN priors in handling
diverse natural images (Rombach et al. 2021; Ramesh et al. 2022; Saharia et
al. 2022b). Kawar et al. (Kawar et al. 2023) applied complex text-guided
semantic edits to real images. ControlNet (Zhang and Agrawala 2023) enables
conditional inputs, such as edge maps, segmentation maps, keypoints, etc., to
T2I models. Liu et al. (Liu et al. 2023) and Wang et al. (Wang et al. 2023)
respectively utilized generative diffusion priors to image colorization and
super-resolution.

## Pixel-Aware Stable Diffusion Network

Our method is based on generative diffusion priors. In particular, we utilize
the powerful pre-trained SD (Rombach et al. 2021) model, while alternative
diffusion models such as DALLE2 (Ramesh et al. 2022) and Imagen (Saharia et
al. 2022b) can also be adopted. The architecture of our pixel-aware SD (PASD)
network is depicted in Fig. 2. One can see that in addition to the pre-trained
SD model, PASD has three main modules: a degradation removal module to extract
degradation insensitive low-level control features, a high-level information
extraction module to extract semantic control features, and a pixel-aware
cross-attention (PACA) module to perform pixel-level guidance for diffusion.
While our PASD is mainly designed for the Real-ISR task, it can be readily
used for personalized stylization by simply switching the base diffusion model
to a personalized one.

### Degradation Removal Module

Real-world LQ images usually suffer from complex and unknown degradations. We
thus employ a degradation removal module to reduce the impact of degradations
and extract “clean” features from the LQ image to control the diffusion
process. As shown in Fig. 2, we adopt a pyramid network to extract multi-scale
feature maps with 1/2, 1/4 and 1/8 scaled resolutions of the input LQ image.
Intuitively, it is anticipated that these features can be used to approximate
the HQ image at the corresponding scale as close as possible so that the
subsequent diffusion module could focus on recovering realistic image details,
alleviating the burden of distinguishing image degradations. Therefore, we
introduce an intermediate supervision by employing a convolution layer “toRGB”
to turn every single-scale feature maps into the HQ RGB image space. We apply
an $L_{1}$ loss on each resolution scale to force the reconstruction at that
scale to be close to the pyramid decomposition of the HQ image:
$\mathcal{L_{DR}}=\sum_{s}||\mathbf{I}_{hq}^{s}-\mathbf{I}_{sr}^{s}||_{1}$,
where $\mathbf{I}_{hq}^{s}$ and $\mathbf{I}_{sr}^{s}$ represent the HQ ground-
truth and ISR output at scale $s$. Note that $L_{DR}$ is only required in the
Real-ISR task.

Table 1: The PSNR, SSIM, FID, CLIP-FID, LPIPS, DISTS and MUSIQ indices of
different Real-ISR models on both synthesized (DIV2K) and real-world (RealSR
and DRealSR) test datasets.

Datasets | Metrics | BSRGAN | Real-ESRGAN | LDL | FeMaSR | SwinIR-GAN | LDM | SD Upscaler | StableSR | PASD
---|---|---|---|---|---|---|---|---|---|---
DIV2K valid | PSNR$\uparrow$ | 23.4105 | 23.1465 | 22.7449 | 21.8552 | 22.6459 | 21.4763 | 21.2083 | 20.8778 | 21.8530
SSIM$\uparrow$ | 0.6078 | 0.6212 | 0.6172 | 0.5426 | 0.6051 | 0.5572 | 0.5466 | 0.5256 | 0.5215
FID$\downarrow$ | 92.0801 | 85.8453 | 91.0641 | 83.7743 | 85.2116 | 93.0184 | 92.1536 | 69.5598 | 68.7851
CLIP-FID$\downarrow$ | 13.3123 | 13.1279 | 14.1164 | 10.6802 | 13.0361 | 14.0566 | 12.9387 | 9.9444 | 10.3318
LPIPS$\downarrow$ | 0.4263 | 0.4030 | 0.4160 | 0.4100 | 0.4055 | 0.4497 | 0.4302 | 0.4376 | 0.4304
DISTS$\downarrow$ | 0.1787 | 0.1691 | 0.1770 | 0.1711 | 0.1692 | 0.1844 | 0.1747 | 0.1761 | 0.1680
MUSIQ$\uparrow$ | 55.9025 | 57.2905 | 56.3724 | 59.0301 | 56.8953 | 40.2183 | 63.4425 | 48.1331 | 65.8658
RealSR | PSNR$\uparrow$ | 26.7457 | 26.0231 | 25.6422 | 25.4993 | 26.2911 | 24.6439 | 25.871 | 26.0326 | 24.8065
SSIM$\uparrow$ | 0.7767 | 0.7742 | 0.7696 | 0.7518 | 0.7816 | 0.6468 | 0.6430 | 0.7714 | 0.6947
FID$\downarrow$ | 62.7142 | 66.5168 | 72.3305 | 62.8607 | 63.8406 | 61.8616 | 67.2451 | 70.2201 | 57.7270
CLIP-FID$\downarrow$ | 7.449 | 7.8258 | 9.1593 | 6.3895 | 8.2282 | 11.9536 | 9.2767 | 8.3158 | 7.2620
LPIPS$\downarrow$ | 0.2674 | 0.2709 | 0.2761 | 0.2961 | 0.2591 | 0.3967 | 0.3079 | 0.2617 | 0.2926
DISTS$\downarrow$ | 0.1354 | 0.1431 | 0.1504 | 0.1328 | 0.1345 | 0.1656 | 0.1603 | 0.1382 | 0.1263
MUSIQ$\uparrow$ | 63.9907 | 62.6299 | 63.4630 | 61.4377 | 65.0789 | 51.4555 | 62.7663 | 67.2719 | 67.9338
DRealSR | PSNR$\uparrow$ | 28.3408 | 27.9152 | 27.7215 | 26.5862 | 27.8456 | 24.3063 | 26.4961 | 27.5304 | 25.0244
SSIM$\uparrow$ | 0.8205 | 0.8247 | 0.8333 | 0.7683 | 0.8206 | 0.6556 | 0.7698 | 0.8303 | 0.6943
FID$\downarrow$ | 19.7078 | 23.1816 | 25.6747 | 19.5665 | 24.6497 | 27.0092 | 24.2128 | 18.2130 | 19.9174
CLIP-FID$\downarrow$ | 3.6615 | 3.6877 | 3.7812 | 3.3268 | 3.6527 | 7.2814 | 5.4362 | 3.4574 | 3.4227
LPIPS$\downarrow$ | 0.2929 | 0.2818 | 0.2785 | 0.3374 | 0.2838 | 0.4348 | 0.4055 | 0.2750 | 0.2739
DISTS$\downarrow$ | 0.0870 | 0.0901 | 0.0962 | 0.0994 | 0.0925 | 0.1259 | 0.1024 | 0.0853 | 0.0958
MUSIQ$\uparrow$ | 55.0083 | 54.8643 | 54.9784 | 56.3682 | 55.3148 | 54.4335 | 50.0446 | 51.9831 | 63.2654

### Pixel-Aware Cross Attention (PACA)

The main challenge of utilizing pre-trained T2I diffusion priors for image
restoration tasks lies in how to enable the diffusion process be aware of
image details and textures in pixel-level. The well-known ControlNet can
support task-specific conditions (e.g., edges, segmentation masks) well but
fail for pixel-level control. Given a feature map
$\textbf{x}\in\mathbb{R}^{h\times w\times c}$ from U-Net, where $\\{h,w,c\\}$
are feature height, width and channel numbers, and a skipped feature map
$\textbf{y}\in\mathbb{R}^{h\times w\times c}$ from ControlNet, Zhang and
Agrawala (Zhang and Agrawala 2023) proposed a unique type of convolution layer
$\mathcal{Z}$ called “zero convolution” to connect them:

$\tilde{\mathbf{x}}=\mathbf{x}+\mathcal{Z}(\mathbf{y}),$ (1)

where $\tilde{\mathbf{x}}$ is the output feature map. The zero convolution is
easy-to-implement. However, simply adding the feature maps from the two
networks may fail to pass pixel-level precise information, leading to
structure inconsistency between the input LQ and output HQ images. Fig. 1
shows an example. One can see that by simply applying ControlNet to the LQ
input, there are obvious structure inconsistencies in areas such as leaves,
flowers, moustaches and glasses.

To address this problem, some methods employ a skipped connection outside the
U-Net (Wang et al. 2023) to add image details. However, this introduces
additional training in image feature domain, and limits the application of the
trained network to tasks performed in latent space (e.g., image stylization).
In this work, we introduce a simple pixel-aware cross attention (PACA) to
solve this issue. We reshape $\mathbf{x}$ and $\mathbf{y}$ to
$\mathbf{x}^{\prime}\in\mathbb{R}^{h*w\times c}$ and
$\mathbf{y}^{\prime}\in\mathbb{R}^{h*w\times c}$, and consider
$\mathbf{y}^{\prime}$ as the context input. The PACA (see the brown-colored
block in Fig. 2) can be computed as follows:

$PACA(\mathbf{Q},\mathbf{K},\mathbf{V})=Softmax(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}})\cdot\mathbf{V},$
(2)

where $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$ are calculated according to
operations $to\\_q(\mathbf{x}^{\prime}),to\\_k(\mathbf{y}^{\prime})$ and
$to\\_v(\mathbf{y}^{\prime})$, respectively.

The conditional feature input $\mathbf{y}^{\prime}$ is of length $h*w$, which
equals to the total number of pixels of latent feature $\mathbf{x}$. Since
feature $\mathbf{y}^{\prime}$ has not been converted into the latent space by
the Encoder, it preserves well the original image structures. Therefore, our
PASD model can manage to perceive pixel-wise information from the conditional
input $\mathbf{y}^{\prime}$ via PACA. As can be seen in Fig. 8, with the help
of PACA, the output of our PASD network can reproduce realistic and faithful
image structures and textures in pixel-level.

Figure 3: Realistic image super-resolution results by different methods on
test images. Figure 4: User study results of different Real-ISR methods.

### High-Level Information

Our method is based on the pre-trained SD model where text is used as the
input, while Real-ISR typically takes the LQ image as the input. Though some
SD-based Real-ISR methods (Wang et al. 2023) employ the null-text prompt, it
has been demonstrated that content-related captions could improve the
synthesis results (Rombach et al. 2021). As shown in Fig. 2, we employ the
pre-trained ResNet (He et al. 2016), YOLO (Redmon et al. 2016) and BLIP (Li et
al. 2023) networks to extract image classification, object detection and image
caption information from the LQ input, and employ the CLIP (Radford et al.
2021) encoder to convert the text information into image-level features,
providing additional semantic signal to control the diffusion process.

The classifier-free guidance (Ho and Salimans 2021) technique is adopted here:

$\tilde{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})=\mathbf{\epsilon}(\mathbf{z}_{t},\mathbf{c})+\omega\mathbf{\epsilon}(\mathbf{z}_{t},\mathbf{c}_{neg}),$
(3)

where $\tilde{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})$ and
$\mathbf{\epsilon}(\mathbf{z}_{t},\mathbf{c}_{neg})$ are conditional and
unconditional $\mathbf{\epsilon}$-predictions (Ho, Jain, and Abbeel 2020).
$\mathbf{c}$ and $\mathbf{c}_{n}$ are respectively the positive and negative
text prompts, $\mathbf{z}_{t}$ is the latent feature at step $t$, and $\omega$
adjusts the guidance scale. The unconditional $\mathbf{\epsilon}$-prediction
$\mathbf{\epsilon}(\mathbf{z}_{t},\mathbf{c}_{neg})$ can be achieved with
negative prompts. In practice, we empirically combine words like “noisy”,
“blurry”, “low resolution”, etc., as negative prompts. The negative prompts
play a key role to trade off mode coverage and sample quality during
inference. It is optional but could boost much the Real-ISR performance.

### Personalized Stylization

Our method is primarily designed for Real-ISR, which can be considered as an
image-to-image translation problem (i.e., from LQ images to HQ images). Thanks
to the proposed PACA, the translation can be done in pixel-level. Inspired by
the recent work of AnimateDiff (Guo et al. 2023), we can replace the base
model of our PASD network with a personalized model during inference (as
illustrated in the top-right corner of Fig. 2) so that it can produce
stylization results. Unlike previous methods (Zhu et al. 2017; Chen, Lai, and
Liu 2018; Chen, Liu, and Chen 2020) that achieve stylization ability by
learning a pixel-to-pixel mapping function using adversarial training, our
PASD approach decouples stylization generation and pixel-to-pixel mapping,
opening a new door for image stylization. By fine-tuning personalized SD
models with a batch of style images or downloading different personalized
models from online communities 111https://civitai.com/,
https://huggingface.co/, one can easily generate various stylized results with
our PASD method.

Table 2: The FID, CLIP-FID and MUSIQ indices of different stylization models
on test data.

Datasets | Metrics | CartoonGAN | AnimeGAN | DCT-Net | Instruct-Pix2Pix | SD image2image | ControlNet | PASD
---|---|---|---|---|---|---|---|---
FFHQ | FID$\downarrow$ | 110.6474 | 121.8184 | 122.6714 | 108.0571 | 130.0608 | 98.9197 | 106.4948
CLIP-FID$\downarrow$ | 53.7454 | 58.4010 | 50.6869 | 39.3259 | 63.0608 | 37.9589 | 37.6698
MUSIQ$\uparrow$ | 71.9848 | 70.3372 | 62.6950 | 72.9510 | 74.8963 | 74.7521 | 75.0221
Flicker2K | FID$\downarrow$ | 168.2210 | 177.2038 | 177.8979 | 171.2087 | 163.3093 | 153.4681 | 161.5938
CLIP-FID$\downarrow$ | 72.5560 | 78.4741 | 81.0789 | 71.2098 | 75.5553 | 72.2742 | 70.3800
MUSIQ$\uparrow$ | 71.5768 | 72.7070 | 72.2595 | 74.1627 | 75.9558 | 72.3908 | 73.1320

Figure 5: Stylization (cartoonization) results by different methods on real-
world images. Figure 6: User study results of different stylization methods.

### Training Strategy

Given an HQ image, we first obtain its latent representation $\mathbf{z}_{0}$.
The diffusion algorithm progressively adds noise to the latent image and
yields a noisy latent $\mathbf{z}_{t}$, where $t$ is the diffusion step and is
randomly sampled. Given a number of conditions including diffusion step $t$,
LQ input $\mathbf{I}_{lq}$ and text prompt $\mathbf{c}$, we learn a PASD
network $\mathbf{\epsilon}_{\theta}$ to predict the noise added to the noisy
latent $\mathbf{z}_{t}$. The optimization objective of the diffusion model is:

$\mathcal{L_{DF}}=\mathbb{E}_{\mathbf{z}_{0},t,\mathbf{c},\mathbf{I}_{lq},\mathbf{\epsilon}\sim\mathcal{N}(0,1)}\big{[}||\mathbf{\epsilon}-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},t,\mathbf{c},\mathbf{I}_{lq})||^{2}_{2}\big{]}.$
(4)

During the training of Real-ISR models, we jointly update the degradation
removal module. The total loss is:

$\mathcal{L}=\mathcal{L_{DF}}+\alpha\mathcal{L_{DR}}$ (5)

where $\alpha$ is a balancing parameter. We empirically set $\alpha=1$.

Fine-tuning all the parameters in the pre-trained SD model would be very
expensive. As in previous works (Zhang and Agrawala 2023; Wang et al. 2023),
we freeze all the parameters in SD, and only train the newly added modules,
including degradation removal module, ControlNet and PACA. The employed
ResNet, YOLO and BLIP and CLIP networks for high-level information extraction
are also fixed. During training, we randomly replace $50\%$ of the text
prompts with null-text prompts. This encourages our PASD model to perceive
semantic contents from input LQ images as a replacement of text prompts.

## Experiments

### Experiment Setup

We adopt the Adam optimizer (Kingma and Ba 2015) to train PASD with a batch
size of $4$. The learning rate is fixed as $5\times 10^{-5}$. The model is
updated for $500K$ iterations with $8$ NVIDIA Tesla 32G-V100 GPUs.

Training and testing dataset. We train PASD on DIV2K (Timofte et al. 2017),
Flickr2K (Agustsson and Timofte 2017), OST (Wang et al. 2018a), and the first
$10000$ face images from FFHQ (Karras, Laine, and Aila 2019). We employ the
degradation pipeline of Real-ESRGAN (Wang et al. 2021b) to synthesize LQ-HQ
training pairs.

In the task of Real-ISR, we evaluate our approach on both synthetic and real-
world datasets. The synthetic dataset is generated from the DIV2K validation
set following the degradation pipeline of Real-ESRGAN (Wang et al. 2021b). For
real-world test dataset, we use two benchmarks, i.e., RealSR (Cai et al. 2019)
and DRealSR (Wei et al. 2020), for evaluation. As for the task of personalized
stylization, we conduct comparisons on the first $100$ face images from FFHQ
as well as the first $100$ images from Flicker2K.

Evaluation metrics. For quantitative evaluation of Real-ISR models, we employ
the widely used perceptual metrics, including FID (Heusel et al. 2017), LPIPS
(Zhang et al. 2018), DISTS (Ding et al. 2022) and MUSIQ (Ke et al. 2021), to
compare the competing Real-ISR models. FID (Heusel et al. 2017) is widely used
to evaluate the image perceptual quality in image generation tasks. It
extracts image features using Inception V3 (Szegedy et al. 2015) trained on
ImageNet. We also adopt a variant of FID, i.e., CLIP-FID (Kynkäänniemi et al.
2023), which uses the CLIP (Radford et al. 2021) features, in evaluation. The
PSN and SSIM indices (evaluated on the Y channel in YCbCr space) are also
reported for reference only because they are not suitable to evaluate
generative models. Since ground-truth images are unavailable in personalized
stylization tasks, we employ FID, CLIP-FID and MUSIQ in evaluation.

For both Real-ISR and stylization tasks, we invite $20$ volunteers to conduct
a user study on $40$ real-world images. Each volunteer is asked to choose the
most preferred one among all the outputs of competing methods, which are
presented to the volunteers in random order.

Figure 7: Real-ISR results by different variants of PASD.

### Experimental Results

Realistic image super-resolution. We compare the proposed PASD method with two
categories of Real-ISR algorithms. The first category is GAN-based methods,
including BSRGAN (Zhang et al. 2021), Real-ESRGAN (Wang et al. 2021b), SwinIR-
GAN (Liang et al. 2021), LDL (Liang, Zeng, and Zhang 2022) and FeMaSR (Chen et
al. 2022). The second category is diffusion-based models, including LDM
(Rombach et al. 2021), SD upscaler (Rombach et al. 2021) and StableSR (Wang et
al. 2023). The quantitative evaluation results on the test data are presented
in Tab. 1, from which we can have the following observations.

First, in term of fidelity measures PSNR/SSIM, the diffusion-based methods are
not advantageous over GAN-based methods. This is because diffusion models have
higher generative capability and hence may synthesize more perceptually
realistic but less “faithful” details, resulting in lower PSNR/SSIM indices.
Second, the diffusion-based methods, especially the proposed PASD, perform
better than GAN-based methods in most perception metrics. This conforms to our
observation on the visual quality (see Fig. 8) of their Real-ISR output.
Third, our proposed PASD achieves the best MUSIQ scores, which is a no-
reference image quality assessment index, on all the three test datasets.

Fig. 8 visualizes the Real-ISR results of competing methods. It can be seen
that our PASD method can generate more realistic details with better visual
quality (see the synthesized textures in fur, flowers, leaves, feathers,
etc.). Fig. 4 presents the results of subjective user study. The proposed PASD
receives the most rank-1 votes, confirming its superiority in generating
realistic image details. More visual results can be found in the supplementary
material.

Personalized Stylization. Similar to the Real-ISR task, we compare the
proposed PASD with two categories of stylization algorithms. The first
category is GAN-based methods, including CartoonGAN (Chen, Lai, and Liu 2018),
AnimeGAN (Chen, Liu, and Chen 2020) and DCT-Net (Men et al. 2022). We re-train
these models with a batch of stylized images generated by a personalized
diffusion model, i.e., ToonYou 222https://civitai.com/models/30240/toonyou.
The second category is diffusion-based algorithms, including InstructPix2Pix
(Brooks, Holynski, and Efros 2023), SD img2img (Rombach et al. 2021) and
ControlNet (Zhang and Agrawala 2023). We replace their base models with the
personalized model for fair comparison.

Tab. 2 shows the quantitative evaluation results. It can be seen that our PASD
method achieves the best or second best results in most indices. Fig. 9 shows
some cartoonization results. One can see that compared with GAN-based methods,
the results of PASD is much cleaner. Compared with the diffusion-based models,
PASD can better preserve image details such as human hair. Due to the limited
space, we only present results with the style of ToonYou here. Please note
that PASD can generate various stylization results by simply switching the
base diffusion model to a personalized one without any additional training
procedure. More stylization results, including the results on image
colorization, can be found in the supplementary materials.

As in the task of Real-ISR, we conduct a user study for subjective assessment
for stylization. Fig. 6 shows the results. Clearly, PASD is preferred by most
subjects.

Table 3: Quantitative results of different variants of PASD on RealSR test
dataset.

Exp. |  Degradation Removal |  High-level info |  Negative prompt | PSNR$\uparrow$ | FID$\downarrow$ | LPIPS$\downarrow$
---|---|---|---|---|---|---
(a) |  | ✓ | ✓ | 23.0590 | 24.9677 | 0.4722
(b) | ✓ |  |  | 26.0218 | 7.8604 | 0.3958
(c) | ✓ | ✓ |  | 25.3468 | 9.1088 | 0.3770
(d) | ✓ |  | ✓ | 24.7121 | 8.5100 | 0.3590
PASD | ✓ | ✓ | ✓ | 24.8065 | 7.2620 | 0.2926

### Ablation Studies

Importance of PACA. We evaluate a variant of PASD by excluding the PACA module
from it, i.e., the features $\mathbf{y}$ extracted from ControlNet are simply
added to features $\mathbf{x}$. As shown in Fig. 7(b), the output becomes
inconsistent with the LQ input in colors and structures, etc. This verifies
the importance of PACA in perceiving pixel-wise local structures.

Role of degradation removal module. To evaluate the effect of degradation
removal module, we remove the “toRGB” modules as well as the pyramid
$\mathcal{L}_{DR}$ loss during model training. As can be seen in Fig. 7(c) and
Tab. 3, removing the degradation removal module leads to dirty outputs and
worse PSNR, FID and LPIPS indices.

Role of high-level information. The high-level information and negative prompt
are optional but very useful for PASD. We simply replace them with null-text
prompt to evaluate their effects. As shown in Fig. 7(d), replacing both high-
level information and negative prompt with null-text prompt results in dirty
outputs with less realistic details, which is also verified by the worse FID
and LPIPS indices in Tab. 3. Abandoning high-level information leads to over-
smoothed results, as illustrated in Fig. 7(e). The output can become dirty
without negative prompt (see Fig. 7(f)). Our full model takes advantages of
both high-level information and negative prompt, and achieves a good balance
between clean-smooth and detailed-dirty outputs (see Fig. 7(g)).

## Conclusion

We proposed a pixel-aware diffusion network, namely PASD, for realistic image
restoration and personalized stylization. By introducing a pixel-aware cross
attention module, PASD succeeded in perceiving image local structures in
pixel-level and achieved robust and perceptually realistic Real-ISR results.
By replacing the base model to a personalized one, PASD could also produce
diverse stylization results with highly consistent semantic contents with the
input. The proposed PASD was simple to implement, and our extensive
experiments demonstrated its effectiveness and flexibility across different
tasks, showing its great potentials for handling complex image restoration and
stylization tasks.

##  Appendix A Appendix

### More Real-ISR Results

In Fig. 8, we show more visual comparisons between our method with state-of-
the-art Real-ISR methods, including Real-ESRGAN (Wang et al. 2021b), SwinIR-
GAN (Liang et al. 2021), LDM (Rombach et al. 2021) and StableSR (Wang et al.
2023). Similar conclusions to the main paper can be made. With the help of
PACA module, our PASD can provide pixel-level guidance on the image
generation, reproducing more realistic fine details and less visual artifacts.

### Various Stylization Results

As mentioned in the main paper, by simply switching the base diffusion model
to a personalized one, our proposed PASD can do various stylization tasks
without any additional training procedure. In the main paper, we have provided
the results by using the ToonYou style. In Fig 9 of this supplementary file,
we show more types of stylization results by using the personalized base
models of Disney 3D, Oil painting and Shinkai. One can see that our PASD
method can keep very well the pixel-wise image details while performing style
transfer.

### Image Colorization

Our PASD can serve as a generic solution for various pixel-wise image-to-image
tasks. In addition to Real-ISR and personalized stylization, we also apply it
to image colorization and show the results in this supplementary file.

Figure 10 shows the qualitative comparisons between PASD and the state-of-the-
art image colorization methods, including DeOldify (Anti 2019), BigColor (Kim
et al. 2022), CT2 (Weng et al. 2022) and DDColor (Kang et al. 2023). One can
see that our PASD generates more photo-realistic and vivid colorization
results. In particular, it significantly alleviates the color bleeding effect,
which often happens in the compared methods.

Figure 8: Realistic image super-resolution results by different methods.
Please zoom-in for better comparison. Figure 9: Stylization results by PASD
with different base models (ToonYou, Disney 3D, Oil painting, Shinkai) on
real-world images. Figure 10: Qualitative comparison of different colorization
methods on real-world images.

## References

  * Agustsson and Timofte (2017) Agustsson, E.; and Timofte, R. 2017.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In _CVPRW_.
  * Anti (2019) Anti, J. 2019.  jantic/deoldify: A deep learning based project for colorizing and restoring old image (and videos!).  Https://github.com/jantic/DeOldify.
  * Brooks, Holynski, and Efros (2023) Brooks, T.; Holynski, A.; and Efros, A. A. 2023.  InstructPix2Pix: Learning to Follow Image Editing Instructions.  In _CVPR_.
  * Cai et al. (2019) Cai, J.; Zeng, H.; Yong, H.; Cao, Z.; and Zhang, L. 2019.  Toward real-world single image super-resolution: A new benchmark and a new model.  In _ICCV_.
  * Chen et al. (2022) Chen, C.; Shi, X.; Qin, Y.; Li, X.; Han, X.; Yang, T.; and Guo, S. 2022.  Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors.  In _ACM MM_.
  * Chen, Liu, and Chen (2020) Chen, J.; Liu, G.; and Chen, X. 2020.  AnimeGAN: A Novel Lightweight GAN for Photo Animation.  In _CVPR_.
  * Chen, Lai, and Liu (2018) Chen, Y.; Lai, Y.-K.; and Liu, Y.-J. 2018.  CartoonGAN: Generative Adversarial Networks for Photo Cartoonization.  In _CVPR_.
  * Dhariwal and Nichol (2021) Dhariwal, P.; and Nichol, A. 2021.  Diffusion Models Beat GANs on Image Synthesis.  In _Arxiv_.
  * Ding et al. (2022) Ding, K.; Ma, K.; Wang, S.; and Simoncelli, E. P. 2022.  Image Quality Assessment: Unifying Structure and Texture Similarity.  _IEEE TPAMI_ , 44.
  * Dong et al. (2014) Dong, C.; Loy, C. C.; He, K.; and Tang, X. 2014.  Image Super-Resolution Using Deep Convolutional Networks.  In _ECCV_.
  * Esser, Rombach, and Ommer (2021) Esser, P.; Rombach, R.; and Ommer, B. 2021.  Taming Transformers for High-Resolution Image Synthesis.  In _CVPR_.
  * Gatys, Ecker, and Bethge (2015) Gatys, L. A.; Ecker, A. S.; and Bethge, M. 2015.  A Neural Algorithm of Artistic Style.  In _Arxiv_.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial networks.  In _NeurIPS_ , 2672–2680.
  * Gu et al. (2019) Gu, J.; Lu, H.; Zuo, W.; and Dong, C. 2019.  Blind super-resolution with iterative kernel correction.  In _CVPR_.
  * Guo et al. (2023) Guo, Y.; Yang, C.; Rao, A.; Wang, Y.; Qiao, Y.; Lin, D.; and Dai, B. 2023.  AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning.  In _ArXiv_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _CVPR_ , 770––778.
  * Hertz et al. (2022) Hertz, A.; Mokady, R.; Tenenbaum, J.; Aberman, K.; Pritch, Y.; and Cohen-Or, D. 2022\.  Prompt-to-prompt image editing with cross attention control.  In _ArXiv_.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017\.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  In _NeurIPS_.
  * Ho et al. (2022) Ho, J.; Chan, W.; Saharia, C.; Whang, J.; Gao, R.; Gritsenko, A.; Kingma, D. P.; Poole, B.; Norouzi, M.; Fleet, D. J.; and Salimans, T. 2022.  IMAGEN VIDEO: HIGH DEFINITION VIDEO GENERATION WITH DIFFUSION MODELS.  In _Arxiv_.
  * Ho, Jain, and Abbeel (2020) Ho, J.; Jain, A.; and Abbeel, P. 2020.  Denoising Diffusion Probabilistic Models.  In _NeurIPS_ , 6840–6851.
  * Ho and Salimans (2021) Ho, J.; and Salimans, T. 2021.  Classifier-Free Diffusion Guidance.  In _Arxiv_.
  * Johnson, Alahi, and Li (2016) Johnson, J.; Alahi, A.; and Li, F.-F. 2016.  Perceptual losses for real-time style transfer and super-resolution.  In _ECCV_.
  * Kang et al. (2023) Kang, X.; Yang, T.; Ouyang, W.; Ren, P.; Li, L.; and Xie, X. 2023.  DDColor: Towards Photo-Realistic and Semantic-Aware Image Colorization via Dual Decoders.  In _ICCV_.
  * Karras, Laine, and Aila (2019) Karras, T.; Laine, S.; and Aila, T. 2019.  A Style-Based Generator Architecture for Generative Adversarial Networks.  In _CVPR_.
  * Kawar et al. (2022) Kawar, B.; Elad, M.; Ermon, S.; and Song, J. 2022.  Denoising Diffusion Restoration Models.  In _NIPS_.
  * Kawar et al. (2023) Kawar, B.; Zada, S.; Lang, O.; Tov, O.; Chang, H.; Dekel, T.; Mosseri, I.; and Irani, M. 2023.  Imagic: Text-Based Real Image Editing with Diffusion Models.  In _CVPR_.
  * Ke et al. (2021) Ke, J.; Wang, Q.; Wang, Y.; Milanfar, P.; and Yang, F. 2021.  MUSIQ: Multi-scale Image Quality Transformer.  In _ICCV_.
  * Kim et al. (2022) Kim, G.; Kang, K.; Kim, S.; Lee, H.; Kim, S.; Kim, J.; Baek, S.-H.; and Cho, S. 2022\.  BigColor: Colorization using a Generative Color Prior for Natural Images.  In _ECCV_.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In _Arxiv_.
  * Kumari et al. (2023) Kumari, N.; Zhang, B.; Zhang, R.; Shechtman, E.; and Zhu, J.-Y. 2023.  Multi-Concept Customization of Text-to-Image Diffusion.  In _CVPR_.
  * Kynkäänniemi et al. (2023) Kynkäänniemi, T.; Karras, T.; Aittala, M.; Aila, T.; and Lehtinen, J. 2023.  The Role of ImageNet Classes in Fréchet Inception Distance.  In _ICLR_.
  * Lai et al. (2017) Lai, W.-S.; Huang, J.-B.; Ahuja, N.; and Yang, M.-H. 2017.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution.  In _CVPR_.
  * Ledig et al. (2017) Ledig, C.; Theis, L.; Huszar, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; and Shi, W. 2017.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.  In _CVPR_.
  * Li et al. (2023) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  In _ICML_.
  * Li et al. (2017) Li, Y.; Fang, C.; Yang, J.; Wang, Z.; Lu, X.; and Yang, M.-H. 2017.  Universal Style Transfer via Feature Transforms.  In _NeurIPS_.
  * Liang et al. (2021) Liang, J.; Cao, J.; Sun, G.; Zhang, K.; Gool, L. V.; and Timofte, R. 2021.  SwinIR: Image Restoration Using Swin Transformer.  _ArXiv_.
  * Liang, Zeng, and Zhang (2022) Liang, J.; Zeng, H.; and Zhang, L. 2022.  Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution.  In _CVPR_.
  * Lim et al. (2017) Lim, B.; Son, S.; Kim, H.; Nah, S.; and Lee, K. M. 2017.  Enhanced Deep Residual Networks for Single Image Super-Resolution.  In _CVPRW_.
  * Liu et al. (2023) Liu, H.; Xing, J.; Xie, M.; Li, C.; and Wong, T.-T. 2023.  Improved Diffusion-based Image Colorization via Piggybacked Models.  In _ArXiv_.
  * Lu et al. (2022) Lu, C.; Zhou, Y.; Bao, F.; Chen, J.; Li, C.; and Zhu, J. 2022.  DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps.  In _NeurIPS_.
  * Lugmayr, Danelljan, and Timofte (2019) Lugmayr, A.; Danelljan, M.; and Timofte, R. 2019.  Unsupervised Learning for Real-World Super-Resolution.  In _ICCVW_.
  * Men et al. (2022) Men, Y.; Yao, Y.; Cui, M.; Lian, Z.; and Xie, X. 2022.  DCT-Net: Domain-Calibrated Translation for Portrait Stylization.  In _ACM TOG_.
  * Meng et al. (2023) Meng, C.; Rombach, R.; Gao, R.; Kingma, D. P.; Ermon, S.; Ho, J.; and Salimans, T. 2023.  On Distillation of Guided Diffusion Models.  In _CVPR_.
  * Qin et al. (2023) Qin, C.; Zhang, S.; Yu, N.; Feng, Y.; Yang, X.; Zhou, Y.; Wang, H.; Niebles, J. C.; Xiong, C.; Savarese, S.; Ermon, S.; Fu, Y.; and Xu, R. 2023.  UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild.  In _ArXiv_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  In _ICML_.
  * Ramesh et al. (2022) Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022.  Hierarchical Text-Conditional Image Generation with CLIP Latents.  In _Arxiv_.
  * Redmon et al. (2016) Redmon, J.; Divvala, S.; Girshick, R.; and Farhadi, A. 2016.  You Only Look Once: Unified, Real-Time Object Detection.  In _CVPR_.
  * Rombach et al. (2021) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2021.  High-Resolution Image Synthesis with Latent Diffusion Models.  In _CVPR_.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-Resolution Image Synthesis with Latent Diffusion Models.  In _CVPR_.
  * Ruiz et al. (2023) Ruiz, N.; Li, Y.; Jampani, V.; Pritch, Y.; Rubinstein, M.; and Aberman, K. 2023\.  DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation.  In _CVPR_.
  * Saharia et al. (2022a) Saharia, C.; Chan, W.; Chang, H.; Lee, C. A.; Ho, J.; Salimans, T.; Fleet, D.; and Norouzi, M. 2022a.  Palette: Image-to-Image Diffusion Models.  In _Arxiv_.
  * Saharia et al. (2022b) Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E.; Ghasemipour, S. K. S.; Ayan, B. K.; Mahdavi, S. S.; Lopes, R. G.; Tim Salimans, J. H.; Fleet, D. J.; and Norouzi, M. 2022b.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.  In _Arxiv_.
  * Song, Meng, and Ermon (2021) Song, J.; Meng, C.; and Ermon, S. 2021.  Denoising diffusion implicit models.  In _ICLR_.
  * Szegedy et al. (2015) Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015.  Going Deeper with Convolutions.  In _CVPR_.
  * Timofte et al. (2017) Timofte, R.; Agustsson, E.; Gool, L. V.; Yang, M.-H.; and Zhang, L. 2017.  Ntire 2017 challenge on single image super-resolution: Methods and results.  In _CVPRW_ , 114–125.
  * Wang et al. (2023) Wang, J.; Yue, Z.; Zhou, S.; Chan, K. C.; and Loy, C. C. 2023.  Exploiting Diffusion Prior for Real-World Image Super-Resolution.  In _Arxiv_.
  * Wang et al. (2021a) Wang, X.; Li, Y.; Zhang, H.; and Shan, Y. 2021a.  Towards Real-World Blind Face Restoration with Generative Facial Prior.  In _CVPR_.
  * Wang et al. (2021b) Wang, X.; Xie, L.; Dong, C.; and Shan, Y. 2021b.  Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data.  In _ICCVW_.
  * Wang et al. (2018a) Wang, X.; Yu, K.; Dong, C.; and Loy, C. C. 2018a.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In _CVPR_.
  * Wang et al. (2018b) Wang, X.; Yu, K.; Wu, S.; Gu, J.; Liu, Y.; Dong, C.; Qiao, Y.; and Loy, C. C. 2018b.  ESRGAN: Enhanced super-resolution generative adversarial networks.  In _ECCVW_.
  * Wei et al. (2020) Wei, P.; Xie, Z.; Lu, H.; Zhan, Z.; Ye, Q.; Zuo, W.; and Lin, L. 2020.  Component Divide-and-Conquer for Real-World Image Super-Resolution.  In _ECCV_.
  * Weng et al. (2022) Weng, S.; Sun, J.; Li, Y.; and Li, S. 2022.  CT2: Colorization transformer via color tokens.  In _ECCV_.
  * Yang et al. (2022a) Yang, B.; Gu, S.; Zhang, B.; Zhang, T.; Chen, X.; Sun, X.; Chen, D.; and Wen, F. 2022a.  Paint by Example: Exemplar-based Image Editing with Diffusion Models.  In _ArXiv_.
  * Yang et al. (2008) Yang, J.; Wright, J.; Huang, T. S.; and Ma, Y. 2008.  Image super-resolution as sparse representation of raw image patches.  In _CVPR_.
  * Yang et al. (2021) Yang, T.; Ren, P.; Xie, X.; ; and Zhang, L. 2021.  GAN Prior Embedded Network for Blind Face Restoration in the Wild.  In _CVPR_.
  * Yang et al. (2022b) Yang, T.; Ren, P.; Xie, X.; Hua, X.; and Zhang, L. 2022b.  Beyond a Video Frame Interpolator: A Space Decoupled Learning Approach to Continuous Image Transition.  In _ECCVW_.
  * Zhang et al. (2021) Zhang, K.; Liang, J.; Gool, L. V.; and Timofte, R. 2021.  Designing a Practical Degradation Model for Deep Blind Image Super-Resolution.  In _ICCV_ , 4791–4800.
  * Zhang and Agrawala (2023) Zhang, L.; and Agrawala, M. 2023.  Adding Conditional Control to Text-to-Image Diffusion Models.  In _NeurIPS_.
  * Zhang et al. (2018) Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.  In _CVPR_.
  * Zhang et al. (2023) Zhang, Y.; Huang, N.; Tang, F.; Huang, H.; Ma, C.; Dong, W.; and Xu, C. 2023.  Inversion-Based Style Transfer with Diffusion Models.  In _CVPR_.
  * Zhou et al. (2022) Zhou, S.; Chan, K. C.; Li, C.; and Loy, C. C. 2022.  Towards Robust Blind Face Restoration with Codebook Lookup Transformer.  In _NeurIPS_.
  * Zhu et al. (2017) Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks.  In _ICCV_.


# Outlier-robust Kalman filters with mixture correntropy

Hongwei Wang  Wei Zhang  Junyi Zuo  Heping Wang  National Key Laboratory of
Science and Technology on Communications, University of Electronic Science and
Technology of China, Chengdu, People’s Republic of China, 611731  School of
Aeronautics, Northwestern Polytechnical University, Xi’an, People’s Republic
of China, 710072

###### Abstract

We consider the robust filtering problem for a nonlinear state-space model
with outliers in measurements. To improve the robustness of the traditional
Kalman filtering algorithm, we propose in this work two robust filters based
on mixture correntropy, especially the double-Gaussian mixture correntropy and
Laplace-Gaussian mixture correntropy. We have formulated the robust filtering
problem by adopting the mixture correntropy induced cost to replace the
quadratic one in the conventional Kalman filter for measurement fitting
errors. In addition, a tradeoff weight coefficient is introduced to make sure
the proposed approaches can provide reasonable state estimates in scenarios
where measurement fitting errors are small. The formulated robust filtering
problems are iteratively solved by utilizing the cubature Kalman filtering
framework with a reweighted measurement covariance. Numerical results show
that the proposed methods can achieve a performance improvement over existing
robust solutions.

###### keywords:

Robsut Kalman filter, mixture correntropy, cubature Kalman filter, state
estimation, measurement outliers

††journal:  fundingfundingfootnotetext: This work was supported by the
National Natural Science Foundation of China (NSFC) under Grant 61473227 and
11472222.fundingfundingfootnotetext: Corresponding author: Hongwei Wang
(tianhangxinxiang@163.com).

##  1 Introduction

State estimation for stochastic discrete-time dynamic systems is one of the
vital issues in control engineering, and it has broad applications in various
areas, such as target tracking, sparse signal processing, fault detection and
diagnosis, pose estimation, and many others [1, 2, 3, 4, 5, 6, 7]. The state
estimates of a linear system with Gaussian noises is provided by the
celebrated Kalman filter [8]. For nonlinear systems with a Gaussian assumption
(i.e., both the process and measurement noises are Gaussian), several Kalman-
liked Gaussian approximation filters (GKF) were investigated, e.g., the
unscented Kalman filter [9], cubature Kalman filter [10, 11], to name a few.
These solutions have shown good performance when the Gaussian assumption meets
in systems. In some applications, however, the Gaussian assumption of the
measurement noise may fail since outliers may contaminate measurements due to
unreliable sensors. Outliers lead to the measurement noise having a heavy tail
and becoming non-Gaussian, resulting in substantial degradation of the
existing GKFs.

The sequential Monte-Carlo sampling/particle filter (PF) [12] and the Gaussian
sum filter (GSM) are two general strategies to deal with non-Gaussian noises
caused by measurement outliers. In the PF, a massive number of particles are
involved to approximate the posterior probability density function to obtain
reasonable estimation results. In the GSM, the state estimates are obtained by
combining the results from several parallelly implemented filters via an
interacting procedure. Therefore, both the PF and GSM suffer from a great
computational burden, which prevents them from being widely used in
applications. In addition, a computationally economical approach, i.e.,
integrating the robust cost from $M$-estimation (e.g., Huber’s cost) into the
GKF framework [13, 14, 15], has also been studied. This type of robust filters
was developed by interpreting the GKF filtering problem as linear or nonlinear
regression. Other approaches for robust filtering such as the heavy-tailed
distribution based solution [16] and the $H_{\infty}$ filter [17] were also
reported in the literature.

Recently, a novel local similarity measure called correntropy from
information-theoretic learning is introduced to deal with heavy-tailed non-
Gaussian noises [18, 19, 20], and its associated maximum correntropy criterion
(MCC) has been employed to design a robust filtering algorithm. In [21, 22]
the MCC was first employed to improve the robustness of the KF for linear
systems. Those methods employed the gradient descent approach and ignored the
covariance propagation procedure, which may cause a potential loss of
information. To handle this issue, a robust Kalman filter called MCKF [23] was
developed via recasting the Kalman filtering problem as a linear regression
one. Afterward, several variants of the MCKF were developed for nonlinear
systems [24, 25, 26]. Although the feasibility of the MCC based robust filters
for dealing with non-Gaussian noises has been demonstrated, the default kernel
in the MCC, e.g., the Gaussian kernel, may not sufficient to deal with more
complex data in many practical problems [27, 28]. Besides, there is still no
guideline for the selection of the kernel parameter which has a significant
influence of the MCC associated robust filter.

There is a need, therefore, for designing some kernel parameter-insensitive
algorithms to deal with measurement outliers. To address this challenge, we
propose two robust Kalman filters based on the mixture correntropy. We
formulate the robust filtering problem by utilizing a mixture correntropy
induced loss to replace the quadratic one in the GKF for measurement fitting
errors. In addition, a weighting coefficient is included to seek a tradeoff
between the model and measurement fitting errors. The resulting robust
filtering problem are iteratively solved within the GKF framework with a
reweighted measurement covariance. The simulation results have shown the
superior performance of the proposed algorithms, and also provide a heuristic
rule to design kernel parameters.

The remaining of the paper is organized as follows. In Section 2, we give a
brief introduction of mixture correntropy. In section 3, we formulate the
mixture correntropy based robust Kalman filtering problems and derive the
related algorithms. In Section 4 we present a simulation example to verify the
performance of the proposed algorithm. Finally, Section 5 concludes our work.

_Notation_ : In this paper, boldface lower and upper-case letters represent
column vectors and matrices, respectively. Scalars are denoted by normal font
letters. $A^{T}$ means the transpose of the matrix $A$.
$\mathcal{N}(\cdot,\cdot)$ means a Gaussian distribution. The estimates of
state $\bm{x}_{t}$ given the measurements up to $t=m$ is denoted by
$\hat{\bm{x}}_{t|m}$.

##  2 Brief review of the mixture correntropy

Correntropy is a newly developed similarity measure that originated from
information-theoretic learning. Given two random variables $X$ and $Y$,
correntropy is defined by [18]

$\displaystyle V(X,Y)=E[\kappa(X,Y)]=\int\int{\kappa(X,Y)p(x,y)dxdy}$ (1)

where $\kappa(\cdot,\cdot)$ is a kernel function which satisfies Mercer’s
theorem, $E(\cdot)$ denotes the expectation operation and $p(x,y)$ is the
joint probability density function of $X$ and $Y$. Readers can refer [18] for
the properties of correntropy and its associated maximum correntropy
criterion. The advantages of correntropy in dealing with measurement outliers
in the Kalman filtering framework were illustrated in several literatures,
e.g., [21, 22, 23, 24, 25, 26]. However, correntropy with a single kernel may
suffer performance degradation when dealing with more complex data. In
addition, correntropy is sensitive to the parameter of the kernel function,
which may limit its performance. To address those issues and improve the
flexibility, a mixture correntropy is introduced [27], i.e.,

$\displaystyle
M(X,Y)=E\left(\alpha\kappa_{1}(X,Y)+(1-\alpha)\kappa_{2}(X,Y)\right)$ (2)

where $0<\alpha<1$ is the mixture coefficient, $\kappa_{1}(\cdot)$ and
$\kappa_{2}(\cdot)$ are two different Mercer kernel functions. Calculating the
exact value of the mixture correntropy is in general intractable due to the
lack of knowledge of $p(x,y)$. In practice, only some finite data samples
$\\{x_{i},y_{i}\\}_{i=1}^{N}$ are available, hence the value of mixture
correntropy can be empirically approximated by

$\displaystyle
M(X,Y)=\frac{1}{N}\sum_{i=1}^{N}\left(\alpha\kappa_{1}(e_{i})+(1-\alpha)\kappa_{2}(e_{i}))\right)$
(3)

where $e_{i}=x_{i}-y_{i}$. It is clear that the mixture correntropy will
reduce to the original correntropy when $\alpha=0$ or $\alpha=1$.

###### Remark 1

For simplicity, we mainly focus on the mixture correntropy with two different
kernels in this work. The mixture correntropy, however, has a generalized form
other than the definition in (2), i.e.,
$M(X,Y)=E\left(\sum_{i}\alpha_{i}\kappa_{i}(X,Y)\right)$ with $i\geq 2$,
$\alpha_{i}>0$ and $\sum_{i}\alpha_{i}=1$.

Generally, the difference between $\kappa_{1}(\cdot)$ and $\kappa_{2}(\cdot)$
in the mixture correntropy can be reached via two approaches. In the first
one, $\kappa_{1}(\cdot)$ and $\kappa_{2}(\cdot)$ may come from the same kernel
family but with distinct kernel parameters, resulting in a homogenous mixture
correntropy, e.g., the double-Gaussian kernel mixture correntropy (DG-MC) [27]
where

$\displaystyle\kappa_{1}(e_{i})=\exp(-\frac{e_{i}^{2}}{2\sigma_{1}^{2}})\ ,\
\kappa_{2}(e_{i})=\exp(-\frac{e_{i}^{2}}{2\sigma_{2}^{2}})$

In the other, $\kappa_{1}(\cdot)$ and $\kappa_{2}(\cdot)$ may be the different
types of kernel functions, leading to a heterogenous mixture correntropy,
e.g., the Laplace-Gaussian kernel mixture correntropy (LG-MC) proposed in [28]
in which

$\displaystyle\kappa_{1}(e_{i})=\exp(-\frac{e_{i}^{2}}{2\sigma_{1}^{2}})\ ,\
\kappa_{2}(e_{i})=\exp(-\frac{|e_{i}|}{\sigma_{2}})$

It is apparent that both the DG-MC and LG-MC meet their maximum when $X=Y$
(i.e., two random variables are exactly the same). Therefore, we here define
the mixture correntropy loss in (4) for facilitating the formulation of the
optimization problem

$\displaystyle L(X,Y)=1-M(X,Y)$ (4)

In the next section, we devote to utilizing the DG-MC loss (DG-MCL) and LG-MC
loss (LG-MCL) to design robust filters.

##  3 Derivation of the proposed robust Kalman filter

Consider the stochastic dynamic process described by a state-space model

$\displaystyle\bm{x}_{t}$ $\displaystyle=f(\bm{x}_{t-1})+\bm{w}_{t-1}$ (5)
$\displaystyle\bm{y}_{t}$ $\displaystyle=h(\bm{x}_{t})+\bm{v}_{t}$ (6)

where $\bm{y}_{t}\in\mathcal{R}^{m}$ is a measurement related to the state of
interest $\bm{x}_{t}\in\mathcal{R}^{n}$; $f(\cdot)$ and $h(\cdot)$ are some
known mappings to model the state transition and measurement procedure
respectively; $\bm{w}_{t-1}\sim\mathcal{N}(0,\bm{Q}_{t-1})$ is the process
noise and $\bm{v}_{t}$ is the measurement noise. In canonical Kalman
filtering, $\bm{v}_{t}$ is assumed to be Gaussian, i.e.,
$\bm{v}_{t}\sim\mathcal{N}(0,\bm{R}_{t})$. Under such a Gaussian assumption,
the Kalman filtering problem can be formulated as the following minimization
problem

$\displaystyle\hat{\bm{x}}_{t|t}$ $\displaystyle=\arg\min_{\bm{x}_{t}}\ -\log
p(\bm{x}_{t}|\bm{y}_{1:t})$ $\displaystyle=\arg\min_{\bm{x}_{t}}\left(-\log
p(\bm{x}_{t}|\bm{y}_{1:t-1})-\log p(\bm{y}_{t}|\bm{x}_{t})\right)$ (7)

where $p(\bm{y}_{t}|\bm{x}_{t})$ is the likelihood function given by
$\mathcal{N}(h(\bm{x}_{t}),\bm{R}_{t})$, and $p(\bm{x}_{t}|\bm{y}_{1:t-1})$ is
the predictive density which can be approximated by
$\mathcal{N}(\hat{\bm{x}}_{t|t-1},\bm{P}_{t|t-1})$ in the Gaussian
approximation filtering framework. Substituting both the predictive density
and likelihood distribution into (7), and discarding the terms that do not
depend on $\bm{x}_{t}$, we can rewrite(7) as

$\displaystyle\hat{\bm{x}}_{t|t}$
$\displaystyle=\arg\min_{\bm{x}_{t}}\left(\frac{1}{2}\|\bm{x}_{t}-\hat{\bm{x}}_{t|t-1}\|_{\bm{P}_{t|t-1}^{-1}}^{2}+\frac{1}{2}\|\bm{y}_{t}-h(\bm{x}_{t})\|_{\bm{R}_{t}^{-1}}^{2}\right)$
$\displaystyle=\arg\min_{\bm{x}_{t}}\left(\frac{1}{2}\|\bm{x}_{t}-\hat{\bm{x}}_{t|t-1}\|_{\bm{P}_{t|t-1}^{-1}}^{2}+\frac{1}{2}\sum_{i=1}^{m}e_{t,i}^{2}\right)$
(8)

where $e_{t,i}$ is the $i$-th component of
$\bm{e}_{t}=\bm{R}_{t}^{-1/2}(\bm{y}_{t}-h(\bm{x}_{t}))$. The optimization
problem in (8) can be solved by several GKFs, e.g., CKF. In the following, we
present our robust Kalman filters in conjunction with the CKF which is briefly
introduced in A. It is straightforward to extend the proposed filters with
other GKFs.

From (8) we note that the traditional Kalman filter based on the Gaussian
assumption has a quadratic loss for the measurement fitting error. It is clear
that the quadratic loss is sensitive to outliers, which is the main reason
that causes the performance degradation of the KF in scenarios where
measurement outliers encountered. In order to improve the robustness of the
filtering algorithm against outliers, the DG-MCL and LG-MCL are utilized
separately to replace the quadratic loss for the measurement fitting error to
design the mixture correntropy based robust filters.

###  3.1 DG-MCL based robust Kalman filter

In this section, we first derive a robust Kalman filter based on the DG-MCL.
Adopting the DG-MCL to the measurement fitting error leads to the following
robust filtering problem

$\displaystyle\hat{\bm{x}}_{t|t}=\arg\min_{\bm{x}_{t}}\left\\{\frac{1}{2}\|\bm{x}_{t}-\hat{\bm{x}}_{t|t-1}\|_{\bm{P}_{t|t-1}^{-1}}^{2}+\lambda\left[1-\frac{1}{m}\sum_{i=1}^{m}\left(\alpha\exp(-\frac{e_{i}^{2}}{2\sigma_{1}^{2}})+(1-\alpha)\exp(-\frac{e_{i}^{2}}{2\sigma_{2}^{2}})\right)\right]\right\\}$
(9)

where $\lambda$ is a weighting coefficient to make the balance between the
model fitting error and measurement fitting error. $\lambda$ should be
carefully chosen to obtain a reasonable estimation result. Specifically, we
expect that the performance of the DG-MCL is similar to that of the quadratic
loss when the measurement fitting error is small. It is noticed that for a
small real vale $\delta$, we have

$\displaystyle e^{\delta}\approx 1+\delta$

Therefore, for a small measurement fitting error, the DG-MCL can be
approximated as

$\displaystyle{L}_{DG-
MCL}\approx\frac{\alpha\sigma_{2}^{2}+(1-\alpha)\sigma_{1}^{2}}{m\sigma_{1}^{2}\sigma_{2}^{2}}\frac{1}{2}\sum_{i=1}^{m}{e_{t,i}^{2}}$

In order to maintain the similarity of the quadratic loss and DG-MCL when the
measurement fitting error is small, $\lambda$ should be determined as

$\displaystyle\lambda=\frac{m\sigma_{1}^{2}\sigma_{2}^{2}}{\alpha\sigma_{2}^{2}+(1-\alpha)\sigma_{1}^{2}}$
(10)

Differencing the cost function in (9) with regards to $\bm{x}_{t}$, we have

$\displaystyle\bm{P}_{t|t-1}^{-1}(\bm{x}_{t}-\hat{\bm{x}}_{t|t-1})-\frac{\lambda}{m}\sum_{i=1}^{m}\left(\alpha\frac{\partial\kappa_{1}(e_{i})}{\partial
e_{i}}\frac{\partial
e_{i}}{\partial\bm{x}_{t}}+(1-\alpha)\frac{\partial\kappa_{2}(e_{i})}{\partial
e_{i}}\frac{\partial e_{i}}{\partial\bm{x}_{t}}\right)=0$ (11)

For the Gaussian kernel, we know that

$\displaystyle\frac{\partial\kappa_{1}(e_{i})}{\partial
e_{i}}=-\frac{e_{i}}{\sigma_{1}^{2}}\kappa_{1}(e_{i}),\quad\frac{\partial\kappa_{2}(e_{i})}{\partial
e_{i}}=-\frac{e_{i}}{\sigma_{2}^{2}}\kappa_{2}(e_{i})$ (12)

Substituting (12) into (11) results in

$\displaystyle\bm{P}_{t|t-1}^{-1}(\bm{x}_{t}-\hat{\bm{x}}_{t|t-1})+\frac{\lambda}{m}\sum_{i=1}^{m}\left(\frac{\alpha\kappa_{1}(e_{i})}{\sigma_{1}^{2}}+\frac{(1-\alpha)\kappa_{2}(e_{i})}{\sigma_{2}^{2}}\right)\frac{e_{i}\partial
e_{i}}{\partial\bm{x}_{t}}=0$ (13)

Define a diagonal matrix $\bm{\Lambda}_{t}$ with its $i$-th element given by

$\displaystyle\Lambda_{t,ii}=\frac{\lambda}{m}\left(\frac{\alpha\kappa_{1}(e_{i})}{\sigma_{1}^{2}}+\frac{(1-\alpha)\kappa_{2}(e_{i})}{\sigma_{2}^{2}}\right)$
(14)

With $\bm{\Lambda}_{t}$, one can rewrite (13) into the matrix format as

$\displaystyle\bm{P}_{t|t-1}^{-1}(\bm{x}_{t}-\hat{\bm{x}}_{t|t-1})+\frac{\partial\bm{e}_{t}}{\partial\bm{x}_{t}}\bm{\Lambda}_{t}\bm{e}_{t}=0$
(15)

Equation (15) is essentially the derivative of the cost function of the
following optimization problem

$\displaystyle\hat{\bm{x}}_{t|t}=\arg\min_{\bm{x}_{t}}\left(\frac{1}{2}\|\bm{x}_{t}-\hat{\bm{x}}_{t|t-1}\|_{\bm{P}_{t|t-1}^{-1}}^{2}+\frac{1}{2}\|\bm{y}_{t}-h(\bm{x}_{t})\|_{\bar{\bm{R}}_{t}^{-1}}^{2}\right)$
(16)

where

$\displaystyle\bar{\bm{R}}_{t}=\bm{R}_{t}^{T/2}\bm{\Lambda}_{t}^{-1}\bm{R}_{t}^{1/2}$
(17)

Despite simple structure, directly solving (16) is intractable due to the fact
that $\bar{\bm{R}}_{t}$ depends on the state $\bm{x}_{t}$ via
$\bm{\Lambda}_{t}$. To address this, we adopt an alternate iterative
algorithm. Specifically, for the given estimate $\hat{\bm{x}}_{t|t}^{k}$ after
the $k$-th iteration, we construct $\bm{\Lambda}^{k}_{t}$ via (14), and then
$\bar{\bm{R}}_{t}^{k}$ via (17). In the next iteration, we solve the
optimization problem (16) with $\bar{\bm{R}}_{t}^{k}$ to obtain
$\hat{\bm{x}}_{t|t}^{k+1}$. It is noted that (16) has a similar structure as
the one under the Gaussian assumption illustrated in (8), which enables us to
solve (16) by applying the existing Gaussian approximation filtering
solutions, e.g., the CKF in A. This iteration loop continues until the
algorithm converges, e.g., for a small tolerance $\epsilon$,

$\displaystyle\|\hat{\bm{x}}_{t|t}^{k+1}-\hat{\bm{x}}_{t|t}^{k}\|<\epsilon$
(18)

At the beginning of the iteration procedure, we initialize $\bm{\Lambda}_{t}$
as an identity matrix, meaning that in the first loop the conventional CKF is
implemented. The proposed robust filter is summarized in Algorithm 1.

Algorithm 1 DG-MCL based robust CKF (DG-MCL-CKF)

Input: $\bm{y}_{1:T}$, $\hat{\bm{x}}_{0|0}$, $\bm{P}_{0|0}$, $\bm{Q}_{1:T}$,
$\bm{R}_{1:T}$,$\sigma_{1}$,$\sigma_{2}$.

Output: $\bm{\hat{}}{\bm{x}}_{t|t}$ and $\bm{P}_{t|t}$ for $t=1:T$.

for $t=1:T$ do

Update $\\{\hat{\bm{x}}_{t|t-1},{\bm{P}}_{t|t-1}\\}$ via {(38),(39)};

Initialize $k=0$, $\bm{\Lambda}_{t}=\bm{I}_{m}$;

repeat $k=1,\cdots,$

Update $\bar{\bm{R}}_{t}$ via (17) using $\bm{\Lambda}_{t}$;

Update $\hat{\bm{x}}_{t|t}^{k}$ and $\bm{P}_{t|t}^{k}$ via (45) and (46)
respectively;

Calculate
$\bm{e}_{t}=\bm{R}_{t}^{-1/2}(\bm{y}_{t}-h(\hat{\bm{x}}_{t|t}^{k}))$, and
update $\bm{\Lambda}_{t}$ via (14);

Calculate $\epsilon=\|\hat{\bm{x}}_{t|t}^{k+1}-\hat{\bm{x}}_{t|t}^{k}\|$;

until $\epsilon<10^{-6}$

$\hat{\bm{x}}_{t|t}=\hat{\bm{x}}_{t|t}^{k}$,
${\bm{P}}_{t|t}={\bm{P}}_{t|t}^{k}$.

end for

###  3.2 LG-MCL based robust Kalman filter

Similarly, the LG-MCL based robust filtering problem can be formulated as

$\displaystyle\hat{\bm{x}}_{t|t}=\arg\min_{\bm{x}_{t}}\Bigg{\\{}\frac{1}{2}\|\bm{x}_{t}$
$\displaystyle-\hat{\bm{x}}_{t|t-1}\|_{\bm{P}_{t|t-1}^{-1}}^{2}+\left[1-\frac{1}{m}\sum_{i=1}^{m}\lambda_{i}\left(\alpha\exp(-\frac{e_{i}^{2}}{2\sigma_{1}^{2}})+(1-\alpha)\exp(-\frac{|e_{i}|}{\sigma_{2}})\right)\right]\Bigg{\\}}$
(19)

where $\lambda_{i}$ is the weighting parameter. The reason why we assign a
weighting parameter to each component of the measurement fitting is that the
kernel functions in the LG-MC are heterogenous. Likewise, for a small fitting
error, the LG-MCL can be approximated by

$\displaystyle{L}_{DG-MCL}\approx\frac{1}{2}\sum_{i=1}^{m}\eta_{i}e_{i,t}^{2}$
(20)

where $\eta_{i}$ is given by

$\displaystyle\eta_{i}=\frac{1}{m}(\frac{\alpha}{\sigma_{1}^{2}}+\frac{2(1-\alpha)}{\sigma_{2}|e_{i}|})$

Therefore, $\lambda_{i}$ should be chosen as in (21) to make sure that the LG-
MCL has a similar performance of the quadratic loss when dealing with the
small measurement fitting error

$\displaystyle\lambda_{i}=\eta_{t}^{-1}=m(\frac{\alpha}{\sigma_{1}^{2}}+\frac{2(1-\alpha)}{\sigma_{2}|e_{i}|})^{-1}$
(21)

Akin to the derivation of the DG-MCL based robust Kalman filter, the
reformulated optimization problem for the LG-MCL based robust filtering
problem is

$\displaystyle\hat{\bm{x}}_{t|t}=\arg\min_{\bm{x}_{t}}\left(\frac{1}{2}\|\bm{x}_{t}-\hat{\bm{x}}_{t|t-1}\|_{\bm{P}_{t|t-1}^{-1}}^{2}+\frac{1}{2}\|\bm{y}_{t}-h(\bm{x}_{t})\|_{\bar{\bm{R}}_{t}^{-1}}^{2}\right)$
(22)

where

$\displaystyle\bar{\bm{R}}_{t}$
$\displaystyle=\bm{R}_{t}^{T/2}\bm{\Lambda}_{t}^{-1}\bm{R}_{t}^{1/2}$ (23)
$\displaystyle\bm{\Lambda}_{t}$
$\displaystyle=\text{diag}(\Lambda_{t,11},\cdots,\Lambda_{t,mm})$ (24)
$\displaystyle\Lambda_{t,ii}$
$\displaystyle=\frac{\lambda_{i}}{m}\left(\frac{\alpha}{\sigma_{1}^{2}}\exp(-\frac{e_{i}^{2}}{2\sigma_{1}^{2}})+\frac{2(1-\alpha)}{|e_{i}|\sigma_{2}}\exp(-\frac{|e_{i}|}{\sigma_{2}})\right)$
(25)

Here we apply the similar iterative procedure to solve (22), and the details
of the resulting robust filter is presented in Algorithm 2.

Algorithm 2 LG-MCL based robust CKF (LG-MCL-CKF)

Input: $\bm{y}_{1:T}$, $\hat{\bm{x}}_{0|0}$, $\bm{P}_{0|0}$, $\bm{Q}_{1:T}$,
$\bm{R}_{1:T}$,$\sigma_{1}$,$\sigma_{2}$.

Output: $\bm{\hat{}}{\bm{x}}_{t|t}$ and $\bm{P}_{t|t}$ for $t=1:T$.

for $t=1:T$ do

Update $\\{\hat{\bm{x}}_{t|t-1},{\bm{P}}_{t|t-1}\\}$ via {(38),(39)};

Initialize $k=0$, $\bm{\Lambda}_{t}=\bm{I}_{m}$;

repeat $k=1,\cdots,$

Update $\bar{\bm{R}}_{t}$ via (23) using $\bm{\Lambda}_{t}$;

Update $\hat{\bm{x}}_{t|t}^{k}$ and $\bm{P}_{t|t}^{k}$ via (45) and (46)
respectively;

Calculate
$\bm{e}_{t}=\bm{R}_{t}^{-1/2}(\bm{y}_{t}-h(\hat{\bm{x}}_{t|t}^{k}))$, and
update $\bm{\Lambda}_{t}$ via (24);

Calculate $\epsilon=\|\hat{\bm{x}}_{t|t}^{k+1}-\hat{\bm{x}}_{t|t}^{k}\|$;

until $\epsilon<10^{-6}$

$\hat{\bm{x}}_{t|t}=\hat{\bm{x}}_{t|t}^{k}$,
${\bm{P}}_{t|t}={\bm{P}}_{t|t}^{k}$.

end for

##  4 Simulations and results

In this section, we analyze the proposed algorithms by investigating two
numerical simulations, e.g., estimating the state of a Van der Pol oscillator
(VPO) and the state-of-charge (SoC) of a battery. For comparison, we also
consider the conventional CKF and some existing robust filters, including the
maximum correntropy derivative-free robust CKF (MCC-CKF) [26], linear
regression and maximum correntropy based CKF (RMCC-CKF) [24] and Huber’s cost
function based CKF (Huber-CKF) [15]. We use two different setups in the MCC-
CKF, i.e., the MCC-CKF1 with $\\{\sigma=100,\ \eta=4\\}$ and MCC-CKF2 with
$\\{\sigma=100,\ \eta=5\\}$ (setting $\sigma=100$ is to deal with the Gaussian
process noise). We set the kernel parameter in the RMCC-CKF to $5$ and the
threshold parameter in the Huber-CKF to $1.345$. In the DG-MCL-CKF and LG-MCL-
CKF, kernel parameters are determined as $\sigma_{1}=4$ and $\sigma_{2}=5$,
and the mixture coefficient $\alpha$ is set to $0.5$.

###### Remark 2

The kernel parameters in the mixture correntropy will influence the
performance of the MCL. In this work, we select these user-defined parameters
by trial and error. Further studies, however, are needed to explore the
detailed parameter-selection strategy, which would be beyond the scope of this
work.

###  4.1 VPO model

The standard VPO model is given by

$\displaystyle\left\\{\begin{array}[]{l}\dot{x}_{1}=x_{2}\\\
\dot{x}_{2}=\mu(1-x_{1}^{2})x_{2}-x_{1}\end{array}\right.$

where $\mu$ is a coefficient to control the nonlinearity of the VPO. Using a
sampling interval $\delta$ to discretize the VPO results

$\displaystyle\bm{x}_{t}=\bm{x}_{t-1}+\left(\begin{array}[]{l}\int_{\delta}x_{2}dt\\\
\int_{\delta}(\mu(1-x_{1}^{2})x_{2}-x_{1})dt\end{array}\right)+\bm{w}_{t-1}$
(28)

where $\bm{x}_{t}=[x_{1,t},x_{2,t}]^{T}$ is the state of interest and
$\bm{w}_{t}$ is the process noise which is assumed to be Gaussian, i.e.,
$\bm{w}_{t}\sim\mathcal{N}(0,\bm{Q}_{t-1})$. We utilize the fourth-order
Runge-Kutta scheme to numerically calculate the integral terms in (28) which
in general have no analytical solutions. Furthermore we assume that the noisy
measurements are gathered via

$\displaystyle y_{t}=(x_{1,t}-1)^{2}+1+v_{t}$

The measurement noise is modeled as the following Gaussian-mixture model to
simulate the heavy-tailed property caused by outliers

$\displaystyle v_{t}\sim(1-\phi)\mathcal{N}(0,R_{t})+\phi\mathcal{N}(0,\varphi
R_{t})$

in which $\phi$ is the contaminating ratio, $\varphi$ is the outlier strength
factor and $R_{t}$ is the covariance of the nominal measurement noise.

In the simulation, we set $\mu=1$, and total samples $T=120$ with the sampling
interval $\delta=0.1$s are involved. The true value of the initial state is
$\bm{x}_{0}=[0,-0.5]^{T}$ and the estimated initial state is generated by a
Gaussian distribution $\mathcal{N}({\bm{x}}_{0},0.01\bm{I}_{2})$. The
covariance of the process noise and the nominal measurement noise are,
respectively, given by $\bm{Q}_{t-1}=0.005\bm{I}_{2}$ and $R_{t}=1$. $L=1000$
Monte Carlo runs are implemented to obtain the simulation results. The time-
averaged root mean square (TRMSE) is employed as a metric, which is defined as

$\displaystyle\text{TRMSE}_{k}=\frac{1}{T}\sum_{t=1}^{T}\sqrt{\frac{1}{L}\sum_{i=1}^{L}(x_{k,t}^{i}-\hat{x}_{k,t}^{i})^{2}},\quad
k=1,2$

First, we have studied the performance of the proposed methods versus the
iteration number. Fig. 1 shows the TRMSEs of $x_{1}$ and $x_{2}$ when the
iteration number of our algorithms varies from $1$ to $10$. It is apparent
that the proposed approaches converge after $2$ or $3$ iterations. In the
following simulations, we set $3$ as a default value of the iteration number
for the proposed algorithms.

Fig. 2 illustrates the TRMSEs of $x_{1}$ and $x_{2}$ with varying $\varphi$
and fixed $\phi=0.2$; Fig. 3 shows these data when $\phi$ varies and
$\varphi=200$. It can be seen that, as expected, the conventional CKF degrades
significantly since the quadratic loss in the CKF is sensitive to outliers.
Overall, our proposed DG-MCL-CKF and LG-MCL-CKF, which have similar
performance, have the smallest TRMSEs among all robust solutions, and the
RMCC-CKF has the largest ones. The inferior performance of the RMCC-CKF is due
primarily to the linearization error during the linear regression procedure.
The Huber-CKF performs comparably against to the MCC-CKF, the performance of
which is significantly influenced by the kernel parameters. This is
illustrated by the fact that the MCC-CKF1 outperforms the MCC-CKF2. Similar
conclusions can also be drawn from Fig. 4 in which we present the RMSE of the
two components of the state for the different algorithms in the scenario where
$\phi=0.3$ and $\varphi=200$.

We have further studied how the parameter $\alpha$ influences the performance
of the proposed method. Table 1 presents the TRMSEs of $x_{1}$ and $x_{2}$ in
the two selected scenarios with different $\alpha$. We only show the data of
the DG-MCL-CKF and omit that of the LG-MCL-CKF due to the similarity.
Obviously, the DG-MCL-CKF degrades to the MCC-CKF1 when $\alpha=1$ while it
turns to be the MCC-CKF2 when $\alpha=0$. From the results one can observe
that the DG-MCL-CKF (i.e., $\alpha\neq 1\ \text{or}\ \alpha\neq 0)$
outperforms both the MCC-CKF1 and MCC-CKF2, so it is concluded that the
mixture correntropy is superior over the conventional correntropy. This may
bring us a heuristic idea for designing a correntropy related robust Kalman
filtering algorithm, i.e., using the mixture correntropy with a larger kernel
parameter and a relative small one to alternate the original correntropy to
skip the kernel parameter selection step. The optimal value of $\alpha$,
however, still needs further investigation.

(a) TRMSE of $x_{1}$ (b) TRMSE of $x_{2}$

Figure 1: TRMSEs versus iteration number when $\varphi=200$ and $\phi=0.1$.

(a) TRMSE of $x_{1}$ (b) TRMSE of $x_{2}$

Figure 2: TRMSEs of different algorithms with varying $\varphi$ with fixed
$\phi=0.2$.

(a) TRMSE of $x_{1}$ (b) TRMSE of $x_{2}$

Figure 3: TRMSEs of different algorithms with varying $\phi$ with fixed
$\varphi=200$.

(a)  (b)

Figure 4: RMSE of $x_{1}$ and $x_{2}$ when $\phi=0.3$ and $\varphi=200$. Table 1: TRMSE of $x_{1}$ and $x_{2}$ with different $\alpha$ $\alpha$ |  | 0(MCC-CKF1) | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1(MCC-CKF2)
---|---|---|---|---|---|---|---|---
$\phi=0.3,\varphi=200$ | $x_{1}$ | 0.4245 | 0.3692 | 0.3625 | 0.3631 | 0.3650 | 0.3578 | 0.4912
$x_{2}$ | 0.4792 | 0.4243 | 0.4180 | 0.4154 | 0.4154 | 0.4127 | 0.5004
$\phi=0.2,\varphi=300$ | $x_{1}$ | 0.2726 | 0.2665 | 0.2599 | 0.2544 | 0.2498 | 0.2458 | 0.3640
$x_{2}$ | 0.3522 | 0.3387 | 0.3327 | 0.3280 | 0.3243 | 0.3213 | 0.3709

###  4.2 SoC estimation in batteries

Owing to its high power density, low cost and long cycle life, the lithium-ion
battery is widely employed in numerous applications such as electric vehicles.
SoC, the level of the amount of charge remaining in a battery, is a crucial
monitored parameter in these applications. Unfortunately, SoC is not in
general physically measurable. A considerable amount of effort has been
devoted to providing an accurate estimate of SoC. One common solution is based
on the Kalman filter, in which the evolution of SoC over time is modeled by a
nonlinear SSM according to the equivalent circuit of a battery.

Figure 5: The equivalent circuit of a lithium-ion battery. $V_{h}$ is a
hysteresis voltage source and $OCV$ is the open circuit voltage source.

Here we consider a equivalent circuit of the lithium-ion battery [29], which
is showed in Fig. 5. The associated nonlinear system is given by

$\displaystyle\left\\{\begin{array}[]{l}\dot{a}=-\beta I\\\
\dot{b}=-\frac{1}{R_{d}C_{d}}b+\frac{I}{C_{d}}\\\ \dot{c}=-\gamma
I\left[0.0755(1-a)+c\right]\\\ y=h(a)-b+c-R_{s}I\\\
h(a)=-1.031e^{-35a}+3.685+0.2156a-0.1178a^{2}+0.3201a^{3}\\\
\end{array}\right.$ (34)

where $a$ is SoC; $b$ is the voltage of the RC circuit; $c$ is the hysteresis
voltage; $I$ is the discharging current; $y$ is the measurement of the
terminal voltage; $\beta$, $R_{d}$, $C_{d}$, and $R_{s}$ are some parameters
of the lithium-ion battery. Clearly, the measurement is complicatedly related
to SoC, hence outlier-contaminated measurements may influence the estimate
accuracy of SoC. We here apply the proposed robust filters to reduce the
negative effect of outliers.

In the simulation, denote $\bm{x}=[a,b,c]^{T}$ and discretize (34) by the
Euler method to construct the SSM so that the KF can be applied to estimate
SoC. We set $\beta=5.634^{-5}$, $R_{d}=3^{-3}$ ${\Omega}$, $C_{d}=9^{3}$ F,
$R_{s}=5^{-3}$ ${\Omega}$, $\gamma=2.47^{-3}$. The process noise obeys
$\mathcal{N}(0,10^{-6}\bm{I}_{3})$, and the measurement noise is from the
following Gaussian mixture noise

$\displaystyle(1-\lambda)\mathcal{N}(0,\bm{R})+\lambda\mathcal{N}(0,\kappa\bm{R})$

where $\bm{R}=10^{-2}$. The true value of the initial state is
$\bm{x}_{0}=[1,0,0]^{T}$. all filters are initialized by
$\mathcal{N}(\hat{\bm{x}}_{0|0},\bm{P}_{0|0})$ where
$\hat{\bm{x}}_{0|0}=[0.95,0.1,0.001]^{T}$ and $\bm{P}_{0|0}=0.05\bm{I}_{3}$.

The TRMSE and RMSE of SoC, which are based on 100 independent Monte Carlo
runs, are utilized as metrics to illustrate the performance of the different
filters. The results for different filters are presented in Fig. 6 and Fig. 7.
The TRMSEs of SoC when $\lambda=0.2$ and $\kappa$ varies are shown in Fig. 6a.
Among the robust filters, the proposed MCL based solutions, which perform
similarly, have the lowest TRMSE for all $\kappa$. It also can be verified
that the TRMSEs of all filters increase slightly when $\kappa$ is small, while
fluctuate dramatically for these larger $\kappa$. Fig. 6b shows the SoC TRMSEs
versus the change of $\lambda$. It is seen that the performance of all robust
filters degrade with the increase of $\lambda$. Again, under such scenarios,
our methods outperform other robust filters.

The RMSEs of SoC for the compared robust solutions under a certain scenario
are presented in Fig. 7. Although all filters have converged over time, the
convergence speed of our methods is faster than others. It is noted that the
convergence values of all robust filters are similar, which are about 30%
smaller than that of the conventional CKF.

(a) Varying $\kappa$ when $\lambda=0.2$

(b) Varying $\lambda$ when $\kappa=10$

Figure 6: TRMSEs of SoC in two different scenarios. Figure 7: RMSEs of SoC
when $\kappa=10$ and $\lambda=0.2$

##  5 Conclusion

In this paper, we have investigated outlier-robust Kalman filters based on
mixture correntropy for a nonlinear system involving the heavy-tailed
measurement noise. Two mixture correntropy induced losses are employed to
replace the quadratic loss for the measurement fitting error in the
conventional Kalman filtering framework. The resulting robust Kalman filtering
problems are then iteratively solved by the conventional CKF with a reweighted
covariance matrix of the measurement noise. It can be noted from the
simulation results that the proposed algorithms can outperform the existing
MCC based solutions.

In the current work, we only consider two kind of Mercer’s kernels, i.e., the
Gaussian kernel and Laplace kernel. We do not take other kernels such as the
Student’s t kernel into account, which can be conducted in the further work.
In addition, the mixture correntropy based on the multi-kernel method are
expected to be the other research direction.

##  Appendix A Cubature Kalman Filter [10]

For the state-space model described in (5) and (6) with the Gaussian process
and measurement noises, the CKF is implemented as follows:

  1. 1.

Initialize the initial state
$\bm{x}_{0}\sim\mathcal{N}(\hat{\bm{x}}_{0|0},\bm{P}_{0|0})$ and generate the
basic weighted cubature point set $\\{\bm{\xi}_{i},\eta_{i}\\}$ for
$i=1,\cdots,2n$, where $n$ is the dimension of the state,
$\bm{\xi}_{i}=\sqrt{n}[\bm{I}]_{i}$, $[\bm{I}]=[\bm{I}_{n},-\bm{I}_{n}]$ and
$\eta_{i}=1/(2n)$.

  2. 2.

Generate the sigma points related to the distribution
$\mathcal{N}(\hat{\bm{x}}_{t-1|t-1},\bm{P}_{t-1|t-1})$

$\displaystyle\bm{P}_{t-1|t-1}$
$\displaystyle=\bm{S}_{t-1|t-1}\bm{S}_{t-1|t-1}^{T}$ (35)
$\displaystyle\bm{\xi}_{i,t-1}$
$\displaystyle=\bm{S}_{t-1|t-1}\bm{\xi}_{i}+\hat{\bm{x}}_{t-1|t-1}$ (36)

  3. 3.

Calculate the predicted state and its associated error covariance

$\displaystyle\bm{\chi}_{i,t-1}=$ $\displaystyle f(\bm{\xi}_{i,t-1})$ (37)
$\displaystyle\hat{\bm{x}}_{t|t-1}=$
$\displaystyle\sum_{i=1}^{2n}\eta_{i}\bm{\chi}_{i,t-1}$ (38)
$\displaystyle\bm{P}_{t|t-1}=$
$\displaystyle\sum_{i=1}^{2n}\eta_{i}(\bm{\chi}_{i,t-1}-\bm{\hat{}}{\bm{x}}_{t|t-1})(\bm{\chi}_{i,t-1}-\bm{\hat{}}{\bm{x}}_{t|t-1})^{T}+\bm{Q}_{t-1}$
(39)

  4. 4.

Generate the sigma points for the predicted distribution
$\mathcal{N}(\hat{\bm{x}}_{t|t-1},\bm{P}_{t|t-1})$

$\displaystyle\bm{P}_{t|t-1}$ $\displaystyle=\bm{S}_{t|t-1}\bm{S}_{t|t-1}^{T}$
(40) $\displaystyle\bm{\phi}_{i,t}$
$\displaystyle=\bm{S}_{t|t-1}\bm{\xi}_{i}+\bm{\hat{}}{\bm{x}}_{t|t-1}$ (41)

  5. 5.

Calculate the predicted measurement, predicted measurement covariance and
state-measurement covariance

$\displaystyle\bm{\psi}_{i,t}$ $\displaystyle=h(\bm{\phi}_{i,t}),\
\bm{\hat{y}}_{t}=\sum_{i=1}^{2n}\eta_{i}\bm{\psi}_{i,t}$ (42)
$\displaystyle\bm{P}_{yy}$
$\displaystyle=\sum_{i=1}^{2n}\eta_{i}\left(\bm{\psi}_{i,t}-\bm{\hat{y}}_{t}\right)\left(\bm{\psi}_{i,t}-\bm{\hat{y}}_{t}\right)^{T}+\bm{R}_{t}$
(43) $\displaystyle\bm{P}_{xy}$
$\displaystyle=\sum_{i=1}^{2n}\eta_{i}\left(\bm{\chi}_{i,t}-\bm{\hat{x}}_{t|t-1}\right)\left(\bm{\psi}_{i,t}-\bm{\hat{y}}_{t}\right)^{T}$
(44)

  6. 6.

Obtain the filtered state and its associated error covariance

$\displaystyle\hat{\bm{x}}_{t|t}$
$\displaystyle=\hat{\bm{x}}_{t|t-1}+\bm{K}_{t}(\bm{y}_{t}-\hat{\bm{y}}_{t})$
(45) $\displaystyle\bm{P}_{t|t}$
$\displaystyle=\bm{P}_{t|t-1}-\bm{K}_{t}\bm{P}_{yy}\bm{K}_{t}^{T}$ (46)
$\displaystyle\bm{K}_{t}$ $\displaystyle=\bm{P}_{xy}\bm{P}_{yy}^{-1}$ (47)

## References

  * [1] M. S. Grewal, A. P. Andrews, Applications of Kalman filtering in aerospace 1960 to the present [historical perspectives], IEEE Control Systems Magazine 30 (3) (2010) 69–78.
  * [2] F. Auger, M. Hilairet, J. M. Guerrero, E. Monmasson, T. Orlowska-Kowalska, S. Katsura, Industrial applications of the Kalman filter: A review, IEEE Transactions on Industrial Electronics 60 (12) (2013) 5458–5471.
  * [3] P. Lu, L. Van Eykeren, E. Van Kampen, C. De Visser, Q. Chu, Adaptive three-step Kalman filter for air data sensor fault detection and diagnosis, Journal of Guidance, Control, and Dynamics (null) (2015) 590–604.
  * [4] H. Wang, H. Yu, M. Hoy, J. Dauwels, H. Wang, Variational Bayesian dynamic compressive sensing, in: 2016 IEEE International Symposium on Information Theory (ISIT), IEEE, 2016, pp. 1421–1425.
  * [5] T.-S. Lou, L. Wang, H. Su, M.-W. Nie, N. Yang, Y. Wang, Desensitized cubature Kalman filter with uncertain parameters, Journal of the Franklin Institute 354 (18) (2017) 8358–8373.
  * [6] H. A. Hashim, L. J. Brown, K. McIsaac, Nonlinear stochastic attitude filters on the special orthogonal group 3: Ito and Stratonovich, IEEE Transactions on Systems, Man, and Cybernetics: Systems.
  * [7] H. A. Hashim, L. J. Brown, K. McIsaac, Nonlinear stochastic position and attitude filter on the special euclidean group 3, Journal of the Franklin Institute 356 (7) (2019) 4144–4173.
  * [8] R. E. Kalman, A new approach to linear filtering and prediction problems, Journal of basic Engineering 82 (1) (1960) 35–45.
  * [9] S. J. Julier, J. K. Uhlmann, Unscented filtering and nonlinear estimation, Proceedings of the IEEE 92 (3) (2004) 401–422.
  * [10] I. Arasaratnam, S. Haykin, Cubature Kalman filters, IEEE Transactions on Automatic Control 54 (6) (2009) 1254–1269.
  * [11] H. Wang, W. Zhang, J. Zuo, H. Wang, Generalized cubature quadrature Kalman filters: derivations and extensions, Journal of Systems Engineering and Electronics 28 (3) (2017) 556–562.
  * [12] M. S. Arulampalam, S. Maskell, N. Gordon, T. Clapp, A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking, IEEE Transactions on signal processing 50 (2) (2002) 174–188.
  * [13] C. D. Karlgaard, H. Schaub, Huber-based divided difference filtering, Journal of Guidance, Control, and Dynamics 30 (3) (2007) 885–891.
  * [14] C. D. Karlgaard, Nonlinear regression Huber–Kalman filtering and fixed-interval smoothing, Journal of Guidance, Control, and Dynamics 38 (2) (2014) 322–330.
  * [15] L. Chang, K. Li, Unified form for the robust Gaussian information filtering based on M-estimate, IEEE Signal Processing Letters 24 (4) (2017) 412–416.
  * [16] H. Wang, H. Li, W. Zhang, H. Wang, Laplace $\ell_{1}$ robust Kalman filter based on majorization minimization, in: 2017 20th International Conference on Information Fusion (Fusion), IEEE, 2017, pp. 1–5.
  * [17] X. Luan, F. Liu, P. Shi, $h_{\infty}$ filtering for nonlinear systems via neural networks, Journal of the Franklin Institute 347 (6) (2010) 1035–1046.
  * [18] W. Liu, P. P. Pokharel, J. C. Príncipe, Correntropy: Properties and applications in non-Gaussian signal processing, IEEE Transactions on Signal Processing 55 (11) (2007) 5286–5298.
  * [19] W. Ma, H. Qu, G. Gui, L. Xu, J. Zhao, B. Chen, Maximum correntropy criterion based sparse adaptive filtering algorithms for robust channel estimation under non-Gaussian environments, Journal of the Franklin Institute 352 (7) (2015) 2708–2727.
  * [20] X. Luo, Y. Xu, W. Wang, M. Yuan, X. Ban, Y. Zhu, W. Zhao, Towards enhancing stacked extreme learning machine with sparse autoencoder by correntropy, Journal of The Franklin Institute 355 (4) (2018) 1945–1966.
  * [21] G. T. Cinar, J. C. Príncipe, Hidden state estimation using the correntropy filter with fixed point update and adaptive kernel size, in: The 2012 International Joint Conference on Neural Networks (IJCNN), IEEE, 2012, pp. 1–6.
  * [22] R. Izanloo, S. A. Fakoorian, H. S. Yazdi, D. Simon, Kalman filtering based on the maximum correntropy criterion in the presence of non-Gaussian noise, in: 2016 Annual Conference on Information Science and Systems (CISS), IEEE, 2016, pp. 500–505.
  * [23] B. Chen, X. Liu, H. Zhao, J. C. Principe, Maximum correntropy Kalman filter, Automatica 76 (2017) 70–77.
  * [24] G. Wang, N. Li, Y. Zhang, Maximum correntropy unscented Kalman and information filters for non-Gaussian measurement noise, Journal of the Franklin Institute 354 (18) (2017) 8659–8677.
  * [25] M. V. Kulikova, Square-root algorithms for maximum correntropy estimation of linear discrete-time systems in presence of non-Gaussian noise, Systems & Control Letters 108 (2017) 8–15.
  * [26] H. Wang, H. Li, W. Zhang, J. Zuo, H. Wang, Maximum correntropy derivative-free robust Kalman filter and smoother, IEEE Access 6 (2018) 70794–70807.
  * [27] B. Chen, X. Wang, N. Lu, S. Wang, J. Cao, J. Qin, Mixture correntropy for robust learning, Pattern Recognition 79 (2018) 318–327.
  * [28] Y. Wang, L. Yang, Q. Ren, A robust classification framework with mixture correntropy, Information Sciences 491 (2019) 306–318.
  * [29] Y. Wang, H. Fang, L. Zhou, T. Wada, Revisiting the state-of-charge estimation for lithium-ion batteries: A methodical investigation of the extended kalman filter approach, IEEE Control Systems Magazine 37 (4) (2017) 73–96.