∎

11institutetext:  Shiqiang Du
11email<EMAIL_ADDRESS>22institutetext:  Key Laboratory of China’s
Ethnic Languages and Information Technology of Ministry of Education, Chinese
National Information Technology Research Institute, Northwest Minzu
University, Lanzhou, Gansu, 730030 China 33institutetext:  College of
Mathematics and Computer Science, Northwest Minzu University, Lanzhou, Gansu,
730030 China 44institutetext:  College of Electronic Engineering, Northwest
Minzu University, Lanzhou, Gansu, 730030 China

# Multi-stage Progressive Reasoning for Dunhuang Murals Inpainting

Wenjie Liu1 Baokai Liu1 Shiqiang Du1,2 Yuqing Shi3 Jiacheng Li1 Jianhua Wang1

(Received: date / Accepted: date)

###### Abstract

Dunhuang murals suffer from fading, breakage, surface brittleness and
extensive peeling affected by prolonged environmental erosion. Image
inpainting techniques are widely used in the field of digital mural
inpainting. Generally speaking, for mural inpainting tasks with large area
damage, it is challenging for any image inpainting method. In this paper, we
design a multi-stage progressive reasoning network (MPR-Net) containing global
to local receptive fields for murals inpainting. This network is capable of
recursively inferring the damage boundary and progressively tightening the
regional texture constraints. Moreover, to adaptively fuse plentiful
information at various scales of murals, a multi-scale feature aggregation
module (MFA) is designed to empower the capability to select the significant
features. The execution of the model is similar to the process of a mural
restorer (i.e., inpainting the structure of the damaged mural globally first
and then adding the local texture details further). Our method has been
evaluated through both qualitative and quantitative experiments, and the
results demonstrate that it outperforms state-of-the-art image inpainting
methods.

###### Keywords:

Image inpainting Multi-stage progressive network Multi-scale feature
aggregation Dunhuang murals

##  1 Introduction

Dunhuang murals refer to the paintings on the interior walls of the Dunhuang
Caves in China, renowned for their superb technique and magnificent scope,
which are valuable material cultural heritage. Dunhuang murals have profound
historical origins, spanning multiple dynasties, including the Western Wei,
Northern Zhou, Tang, and Song dynasties, and experienced more than a thousand
years of historical vicissitudes. However, the degradation of many murals can
be attributed to natural elements such as radiation, sand, dust, humidity, and
harsh temperatures, which have caused them to become damaged and lose their
original quality, and the different classifications of deteriorated murals are
presented in Fig. 1. As cultural relics are non-renewable resources that will
eventually die out with time, This is a severe loss of human civilization.
Therefore, it is significant to explore effective inpainting techniques for
murals.

In the past, people used manual inpainting to fill in missing areas of murals,
but it has several drawbacks. Firstly, it’s risky because it can damage the
mural surface. Secondly, it’s time-consuming and requires a high level of
skill. Thirdly, it’s costly and irreversible. As a result, the cultural
heritage community is now more careful about using manual inpainting and is
exploring other options like digital inpainting, which uses computer software
to recreate missing areas. Digital inpainting is less risky, more efficient,
and can be corrected easily, so it’s gaining popularity in the restoration of
cultural heritage sites. In this field, Pan et al. Pan and Lu (2003) first
proposed a set of technical solutions for digital mural inpainting from a
macro perspective and described its system architecture and operation
mechanism. Moreover, Flusser et al. Zitová and Flusser (2003) combined
chemical analysis in the mural inpainting process. Wang et al. Qi et al.
(2011) designed the semantic learning inpainting framework for broken human
faces in ancient murals. Criminis et al. Criminisi et al. (2003) introduced
the texture inpainting algorithm based on sample block matching. Yang et al.
Tianshui (2011) improved the priority function of the Criminisi inpainting
algorithm using D-S theory with a data fusion approach and researched the
inpainting of color changes and human-contaminated damages in Dunhuang murals.
Li et al. Li et al. (2018) defined structure-first and texture-later
inpainting strategies to restore broken murals through human-computer
interaction digitally. Liu et al. Liu et al. (2022) designed a novel edge
detector based on self-attention combined with convolution to generate line
drawings of Dunhuang murals. Deep learning-based image inpainting methods have
made significant progress in recent years, but they may face difficulties when
it comes to restoring murals. This is because the process of painting murals
involves creating smooth lines and evocative colors that are different from
those found in natural photographs. Therefore, image inpainting methods that
are designed for photographs may not be well-suited for restoring the unique
edge structure and texture details of a mural. The network splits the
inpainting of the mural into two articulated subtasks, which can infer the
damaged structure and texture from the perspective of global and local
receptive fields. In addition, we construct the Dunhuang murals dataset and
real masks dataset, where the real masks are extracted from the broken regions
of the damaged murals. Therefore, our model can realize practical breakages of
the mutilated murals owning a higher application in specific mural inpainting.
The inpainting performance of our method on realistic damaged murals is shown
in Fig. 2.

(a) Discoloration

(b) Erosion

(c) Breakage

(d) Abscission

Figure 1: Murals of different damage types.

In summary, the main innovations and contributions of this paper are as
follows:

  1. 1.

The multi-stage progressive reasoning network is designed with global to local
receptive fields to solve the problem of the single semantic in recurrent
networks, which achieves both semantically reasonable structures and detail-
rich textures.

  2. 2.

To effectively utilize the features inferred from the model, a multi-scale
feature aggregation module is designed to empower the capability of dynamic
selection from the significant semantics, leading to semantic consistency
results with adaptive features.

  3. 3.

We analyze the model in terms of both qualitative and quantitative, which
demonstrate the superiority compared with several state-of-the-art methods in
the Dunhuang murals and benchmark datasets.

Figure 2: The application of inpainting damaged murals.

##  2 Related work

Image inpainting aims to fill the defective areas with realistic content and
ensure consistency regarding global structures and texture details. With the
continuous advancement of research work over the years, significant progress
has been made in image inpainting. Current image inpainting methods can be
divided into traditional diffusion methods and deep learning-based methods.

Traditional inpainting methods: The diffusion-based approach was pioneered by
Bertalmio et al. Bertalmio et al. (2000) used smooth local information to
constrain the inpainting results. Ballester et al. Ballester et al. (2001)
transformed the image inpainting task into a variable differential problem in
terms of the gradient and gray-scale of the image. Tschumperl and Deriche
Tschumperlé and Deriche (2005) presented a general anisotropic diffusion
method with better inpainting results. However, these methods cannot handle
larger missing regions due to the limitations of boundary expansion
prediction. The patch-based image inpainting approaches such as James et al.
(2008); Sun et al. (2005) typically propagated appearance information from the
remained regions or other source images into the missing regions through
various manually defined similarity metrics between patches. Chen et al. Chen
et al. (2023) used multi-scale generative adversarial networks with edge
detection for image inpainting. Since these methods rely on low-dimensional
feature information, they are difficult to achieve satisfactory results in
stable semantic tasks.

Deep learning-based inpainting methods: From the network design perspective,
there can be roughly classified into three types: one-stage, two-stage, and
recurrent networks.

For the one-stage networks, Pathak et al. Pathak et al. (2016) designed a
codec architecture trained by pixel reconstruction loss and adversarial loss.
To improve the consistency of image complementation, Iizuka et al. Iizuka et
al. (2017) introduced global and local discriminators to constrain the
inpainting results. Inspired by several existing approaches Song et al.
(2018); Yan et al. (2018), which own innovative model architecture. Zeng et
al. Zeng et al. (2019) presented the pyramid-context encoder network to
accomplish image complementation by attention transfer. Huang et al. Huang et
al. (2022) proposed a novel region-aware attention module to avoid misleading
invalid information in holes. Due to the limitations of one-stage architecture
lacking sufficient constraints, they occasionally suffer semantic bias and
texture-blurring results.

For the two-stage networks, Yu et al. Yu et al. (2018) raised an improved
generative inpainting network consisting of a coarse network and a refinement
network. A fuzzy inpainting result is obtained from the coarse network, and
the long-term relevance dependence is then simulated in the refinement network
by a contextual attention mechanism. Liu et al. Liu et al. (2019a) used a
fined deep generative model-based approach with a novel coherent semantic
attention layer, which can preserve contextual structures and make more
effective predictions of missing parts. Inspired by Liu et al. (2019b), which
presented an adaptive feature update method, Yu et al. Yu et al. (2019)
promoted irregular mask update by introducing gated convolution. Nazeri et al.
Nazeri et al. (2019) proffered an edge-guided two-stage image inpainting
method by restoring the missing edge map and then combining this edge map with
the incomplete image as the second stage of the input. Due to the limited
structure guidance of edge images, Ren et al. Ren et al. (2019) used smoothed
image edges as the structure representation. Guo et al. Guo et al. (2021)
proposed a novel two-stream network for image inpainting, which models the
structure-constrained texture synthesis and texture-guided structure
reconstruction in a coupled manner. Although the two-stage networks can
achieve better inpainting results, it is prone to the problem of large
training fluctuations and unstable convergence due to their overly deep
network structure.

For recurrent inpainting networks, Xiong et al. Xiong et al. (2019) and Nazeri
et al. Nazeri et al. (2019) filled images with contour edge completion and
image completion in a step-wise manner to ensure structural consistency, Zhang
et al. Zhang et al. (2018) divided the process of image inpainting into four
different stages and used LSTM architectures Hochreiter and Schmidhuber (1997)
to control the information flow of the progressive process. However, those
methods cannot handle irregular holes common in real-world applications. To
address this limitation, Guo et al. Guo et al. (2019) projected a full-
resolution residual network with multiple expansion modules. Li et al. Li et
al. (2019) progressively reconstructed the visual structure, entangling visual
feature reconstruction in image inpainting. Li et al. Li et al. (2020)
followed a recursive framework in the feature space and designed a recurrent
feature inference network with consistent attention. Zeng et al. Zeng et al.
(2020) designed an iterative confidence-based inpainting method. Guo et al.
Guo et al. (2019) offered a full-resolution residual network to fill irregular
holes in the original size images. Oh et al. Oh et al. (2019) proffered the
onion-peel networks for video completion progressively, enabling it to exploit
richer contextual information for the missing regions at every step. However,
the reasoning process of previous recurrent methods generally occurs in
inflexible network architecture lacking awareness of semantics in different
fields. In this paper, a multi-stage progressive reasoning network is designed
to satisfy the requirements from the perspective of global structure and local
texture, which effectively solves this problem.

##  3 Proposed method

The design of MPR-Net is inspired by the inference of LG-Net Quan et al.
(2022) that the global receptive field is more effective in restoring the
structure of an image. In contrast, the local receptive field focuses on
texture detail. According to this conclusion, the model is designed with
global to local receptive fields based on recurrent architecture, as shown in
Fig. 3. In this section, we first introduce the multi-stage progressive
reasoning (MPR) module, then describe the gated feature fusion (GFF) and
multi-scale feature aggregation (MFA) methods, and finally present the
objective function of the network.

Figure 3: The illustration of the proposed MPR-Net. Structure Feature
Reasoning and Texture Feature Reasoning are designed to infer missing content
progressively. After several times of reasoning, structure and texture feature
maps are merged adaptively and then sent to the GFF module for feature
interaction. The fused feature is aggregated in multi-scale and then
deconvoluted to output.

###  3.1 Multi-stage progressive reasoning module

Compared to natural photographs, murals belong to human-drawn images with
unique distinct edges and rich details (i.e., structure and texture features).
The MPR module is designed to model the distinctive semantics of the murals.
Specifically, our MPR module can be divided into three parts:

  1. a.

A multi-stage progressive network containing global to local receptive fields
for dedicated inpainting of unique attributes of the murals.

  2. b.

An attention mechanism of knowledge consistency establishes the connection for
attention scores during recurrent inference.

  3. c.

A feature merging operator for adaptively weighting structure and texture
features.

Inside the module, The attention mechanism acts with every recursive
inference. After filling the damage of the murals, all features generated
during inference are merged with adaptive weighting. In the following, we
elaborate on the execution flow of the MPR module.

####  3.1.1 Multi-stage progressive network

The network consists of two progressive subtasks, where the Structure Feature
Reasoning (SFR) owns a global receptive field that comprehensively considers
the structure features; the Texture Feature Reasoning (TFR) further restores
the missing texture details under the local receptive field. The partial
convolution Liu et al. (2018) is introduced to determine the region which
needs to be updated in the recurrence:

$f_{x,y,z}^{*}=\left\\{\begin{array}[]{ll}W_{z}^{\mathsf{T}}\left(f_{x,y}\odot
m_{x,y}\frac{\operatorname{sum}(1)}{\operatorname{sum}\left(m_{x,y}\right)}\right)+b,&\text{
if }\operatorname{sum}\left(m_{x,y}\right)!=0\\\ 0,&\text{ else
}\end{array}\right.$ (1)

where $f^{*}$ denotes the feature map generated by the partial convolution
layer. $f^{*}_{x,y,z}$ denotes the feature values at $x,y$ locations in the
$z^{th}$ channel. $W_{z}$ is the $z^{th}$ convolution kernel of this layer.
$f_{x,y}$ and $m_{x,y}$ are the feature and mask centered at $x,y$
respectively, and
$\frac{\operatorname{sum}(1)}{\operatorname{sum}\left(m_{x,y}\right)}$ is the
scale factor. The output result is adjusted when the number of convolution
effective input pixels changes. Similarly, the updated mask value at locations
$i,j$ generated by this layer can be formulated as:

$m_{x,y}^{*}=\left\\{\begin{array}[]{ll}1,&\text{if}~{}\operatorname{sum}\left(m_{x,y}\right)!=0\\\
0,&\text{else}\end{array}\right.$ (2)

According to the above formula, a new mask with smaller holes can be received
after partial convolution of the layers. The difference between the updated
and input masks is defined as the area to be repaired in this recurrence. The
updated mask remains original until the subsequent iterations.

####  3.1.2 Knowledge consistent attention

In the image inpainting task, the attention mechanism Li et al. (2020) can
search for features highly relevant to the match from known regions to fill
the missing regions to ensure the semantic consistency of the inpainting
results. In particular, we calculate the similarity between known and missing
region blocks by cosine distance.

${{sim}}_{x,y,x^{\prime},y^{\prime}}^{i}=\left\langle\frac{f_{x,y}}{\left\|f_{x,y}\right\|},\frac{f_{x^{\prime},y^{\prime}}}{\left\|f_{x^{\prime},y^{\prime}}\right\|}\right\rangle$
(3)

where ${{sim}}_{x,y,x^{\prime},y^{\prime}}^{i}$ denotes the correlation
between the masked region pixel at position $(x,y)$ and the available pixel at
position $(x^{\prime},y^{\prime})$ in the $i^{th}$ recurrent. The relevance
score of the pixel at position $(x,y)$ is normalized using the softmax
function, and the attention score map obtained for that pixel is denoted as:

${score}^{\prime}{}_{x,y,x^{\prime},y^{\prime}}^{i}=\frac{\exp\left({sim}{}_{x,y,x^{\prime},y^{\prime}}^{i}\right)}{\sum_{j=1}^{N}\exp\left({sim}{}_{x,y,x^{\prime},y^{\prime}}^{i}\right)}$
(4)

To better fit the designed recursive architecture and weaken the discreteness
caused by the independent attention computation in each recurrent, adaptive
weight values $\lambda$ are used to connect two adjacent recurrences.
Specifically, suppose the pixel at position $(x,y)$ is valid in the last
recurrent. In that case, we adaptively combine the final score of the pixel in
the previous recursion with the score computed in this recursion, as follows:

${score}_{x,y,x^{\prime},y^{\prime}}^{i}=\lambda{score}^{\prime}{}_{x,y,x^{\prime},y^{\prime}}^{i}+(1-\lambda){score}_{x,y,x^{\prime},y^{\prime}}^{i-1}$
(5)

if the pixel value at $(x,y)$ in the previous recursion is invalid, the
attention score obtained in this recursion is the final score:

${score}_{x,y,x^{\prime},y^{\prime}}^{i}={score}^{\prime}{}_{x,y,x^{\prime},y^{\prime}}^{i}$
(6)

Finally, the feature map is reconstructed using the attention scores, and the
new feature map of $(x,y)$ points is calculated as follows:

$\widehat{f}_{x,y}^{i}=\sum_{x^{\prime}\in 1,\ldots W,y^{\prime}\in 1,\ldots
H}{score}_{x,y,x^{\prime},y^{\prime}}^{i}f_{x^{\prime},y^{\prime}}^{i}$ (7)

after the feature map reconstruction is completed, the input features $F$ are
concatenated with the reconstructed feature map $\hat{F}$ and sent into the
pixel convolution layer to obtain the output of this module:

${F^{\prime}}^{i}=\phi(|\widehat{F},F|).$ (8)

####  3.1.3 Adaptive merging operator

Since the structure features in early reasoning fully utilize information from
the damaged region boundary, which should be more deterministic, in contrast,
the inference of texture features lacks sufficient deterministic constraints
since nearing the damaged center, which should own more degrees of freedom.
Therefore, we design an adaptive merging operator to control the weight, and
the $\beta$ is a learnable parameter. The merged structure features
$\boldsymbol{F}_{s}$ and texture feature $\boldsymbol{F}_{t}$ are represented
as:

$\boldsymbol{F}_{s}=\beta\cdot\frac{\sum_{i=1}^{N}f_{x,y,z}^{i}}{\sum_{i=1}^{N}m_{x,y,z}^{i}}$
(9)

$\boldsymbol{F}_{t}=(1-\beta)\cdot\frac{\sum_{i=1}^{M}f_{x,y,z}^{i}}{\sum_{i=1}^{M}m_{x,y,z}^{i}}$
(10)

where $f^{i}$ as the $i^{th}$ feature map generated by the MPR module, and
$f_{x,y,z}$ as the value of the $x,y,z$ position in the feature map $f$.
$m^{i}$ is the binary mask of the feature map $f^{i}$, $N$ and $M$ are the
numbers of recurrent for SFR and TFR, respectively.

###  3.2 Gated feature fusion

Gated feature fusion module Guo et al. (2021) is introduced to further fuse
the inferred structure and texture features, where the soft gating is used to
learn weight rationing from a fully exchanged of the two feature information.
The redundant information brought by multiple iterations can be discarded
based on the fusion operation under feature awareness. The execution flow of
the GFF method is shown in Fig. 4.

Figure 4: Illustration of the Gated Feature Fusion (GFF) module, which
implements interaction between structure and texture features to refine the
results.

Especially, to construct the texture-aware structure features, we denote the
soft gating $\boldsymbol{G}_{t}$ for texture information as:

$\boldsymbol{G}_{t}=\sigma\left(g\left({Concat}\left(\boldsymbol{F}_{t},\boldsymbol{F}_{s}\right)\right)\right)$
(11)

where ${Concat}({\cdot})$ is channel-wise concatenation, $g(\cdot)$ is the
convolution layer of size $3\times 3$, and $\sigma(\cdot)$ is the Sigmoid
activation function. The $\boldsymbol{F}_{t}$ is adaptively superimposed into
the $\boldsymbol{F}_{s}$ by soft gating $\boldsymbol{G}_{t}$:

$\boldsymbol{F}_{s}^{\prime}=\alpha\left(\boldsymbol{G}_{t}\odot\boldsymbol{F}_{t}\right)\oplus\boldsymbol{F}_{s}$
(12)

where $\alpha$ is a training parameter initialized to zero, $\odot$ and
$\oplus$ denote element-wise multiplication and element-wise addition,
respectively. Symmetrically, we calculate the structure-aware texture feature
$\boldsymbol{F}_{t}^{\prime}$ as follows:

$\boldsymbol{G}_{s}=\sigma\left(h\left({Concat}\left(\boldsymbol{F}_{t},\boldsymbol{F}_{s}\right)\right)\right)$
(13)
$\boldsymbol{F}_{t}^{\prime}=\beta\left(\boldsymbol{G}_{s}\odot\boldsymbol{F}_{s}\right)\oplus\boldsymbol{F}_{t}$
(14)

where $h$ follows the equivalent as $g$ and $\beta$ is a training parameter
the same as $\alpha$.

Finally, we concatenate $\boldsymbol{F}_{s}^{\prime}$ and
$\boldsymbol{F}_{t}^{\prime}$ in the channel dimension to obtain the fused
feature $\boldsymbol{F}_{b}$ after gated feature fusion:

$\boldsymbol{F}_{b}={Concat}\left(\boldsymbol{F}_{s}^{\prime},\boldsymbol{F}_{t}^{\prime}\right).$
(15)

###  3.3 Multi-scale feature aggregation

For better extracting beneficial features at various scales in the murals
inpainting task, a multi-scale feature aggregation module is designed with
multiple dilated convolutions for multi-scale semantics collection, as shown
in Fig. 5.

Figure 5: The illustration of the multi-scale feature aggregation (MFA)
module, which owns the ability of dynamic selection from diverse semantic
features of the murals, leading to semantic consistency results with adaptive
features.

$\boldsymbol{F}_{b}^{k}=Conv_{k}\left(\boldsymbol{F}_{b}\right)$ (16)

where $\boldsymbol{F}_{b}$ is the fused feature from GFF module, and
$Conv_{k}$ denotes dilated convolutional layers with expansion rate of $k$,
$k\in\\{1,3,5,7\\}$. We concatenate multi-scale semantic information on the
channel dimension. Then, through a squeeze and excitation operations of SENet
Hu et al. (2018) to capture important channel information.

$\boldsymbol{F}_{c}=SE({Concat}(\boldsymbol{F}_{b}^{1},\boldsymbol{F}_{b}^{3},\boldsymbol{F}_{b}^{5},\boldsymbol{F}_{b}^{7}))$
(17)

The skip connection is used to prevent semantic damage caused by patch-shift
operations. The deconvolution layers are seamlessly embedded into our
architecture to improve computational efficiency. The export
$\boldsymbol{F}_{out}$ of the MFA module is represented as:

$\boldsymbol{F}_{out}=Dconv({Concat}(Dconv(\boldsymbol{F}_{c}),\boldsymbol{F}_{b})).$
(18)

###  3.4 Loss functions

The model is trained with a joint loss, including reconstruction loss,
perceptual loss, style loss, and adversarial loss, to render semantically
reasonable and visually realistic results. In particular, for reconstruction
loss, denote by $\boldsymbol{I}_{gt}$ the ground-truth image, initial binary
mask $\boldsymbol{M}$ (with value 1 for the existing region, 0 otherwise),
$\boldsymbol{I}_{in}=\boldsymbol{I}_{gt}\odot\boldsymbol{M}$ the damaged
image, the output of our generator is defined as $\boldsymbol{I}_{out}$, the
reconstruction loss is defined as:

$\mathcal{L}_{hole}=\frac{1}{N_{\boldsymbol{I}_{gt}}}\left\|(1-\boldsymbol{M})\odot\left(\boldsymbol{I}_{out}-\boldsymbol{I}_{gt}\right)\right\|_{1}$
(19)
$\mathcal{L}_{valid}=\frac{1}{N_{\boldsymbol{I}_{gt}}}\left\|\boldsymbol{M}\odot\left(\boldsymbol{I}_{out}-\boldsymbol{I}_{gt}\right)\right\|_{1}$
(20)

where $N_{\boldsymbol{I}_{gt}}$ denotes the number of elements in
$\boldsymbol{I}_{gt}$, the $\mathcal{L}_{hole}$ and $\mathcal{L}_{valid}$ are
the reconstruction losses on the network output for the hole and the non-hole
pixels respectively.

For capturing structure and texture information during generation learning, we
introduce the perceptual loss, which measures the $l_{1}$ distance between
$\boldsymbol{I}_{gt}$ and $\boldsymbol{I}_{out}$ defined on the pre-trained
VGG-16. Formally, let ${pool}_{i}$ is the feature map of the $i^{th}$ th
pooling layer of VGG-16. $H_{i}$, $W_{i}$, and $C_{i}$ denote the height,
weight, and channel size of the $i^{th}$ th feature map, respectively. The
perceptual loss can be defined as:

$\mathcal{L}_{{perceptual}}=\sum_{i=1}^{N}\frac{1}{H_{i}W_{i}C_{i}}\left|\phi_{{pool}_{i}}^{gt}-\phi_{{pool}_{i}}^{{out}}\right|_{1}$
(21)

We further introduce the style loss to preserve the style coherency. The style
loss is similar to perceptual loss, which calculates as follows:

$\psi_{{pool}_{i}}=\phi_{{pool}_{i}}\phi_{{pool}_{i}}^{\mathsf{T}}$ (22)
$\mathcal{L}_{{style}}=\sum_{i=1}^{N}\frac{1}{C_{i}\times
C_{i}}\left|\frac{1}{H_{i}W_{i}C_{i}}\left(\psi_{\text{pool}_{i}}^{{{gt}}}-\psi_{\text{pool}_{i}}^{{{out}}}\right)\right|_{1}$
(23)

where $\psi_{{pool}_{i}}=\phi_{{pool}_{i}}\phi_{{pool}_{i}}^{\mathsf{T}}$
denotes the gram matrix constructed from the given feature maps. In summary,
our joint loss function is:

$\displaystyle\mathcal{L}_{{joint}}=\lambda_{{hole}}\mathcal{L}_{{hole}}+\lambda_{{valid}}\mathcal{L}_{{valid}}+\lambda_{{perceptual}}\mathcal{L}_{{perceptual}}+\lambda_{{style}}\mathcal{L}_{{style}}$
(24)

for the tradeoff parameters, we empirically set 6 for $\lambda_{{hole}}$, 1
for $\lambda_{{valid}}$, 0.1 for $\lambda_{{perceptual}}$, 180 for
$\lambda_{{style}}$.

##  4 Experiments

In this section, we present the experimental details, datasets, and comparison
methods. Then, extensive experiments are conducted on our Dunhuang murals and
two benchmark datasets for quantitative and qualitative evaluation. Finally,
we perform ablation studies to validate the designed modules and innovative
architecture of our model.

###  4.1 Training setting

The model is implemented using pytorc (1.12.0), with an i9-9700k CPU and
NVIDIA RTX2080ti GPU. Train the model for 400,000 iterations with a learning
rate of 1e-4, and finetuning for 200,000 iterations with a learning rate of
5e-5, using the Adam optimizer for network convergence, all masks and images
are the size of $256\times 256$.

###  4.2 Datasets

Dunhuang murals dataset: We constructed the Dunhuang murals dataset, which
contains a training set of 5000 images and a testing set of 100 images mainly
from the Mus e Guimet in France, the British Museum in the UK, the Dunhuang
Research Institute, and partly obtained from high-resolution scans of Dunhuang
murals publications. Fig. 6 shows different style paintings in the Dunhuang
murals dataset.

(a) Statue painting

(b) Supernatural painting

(c) Sutra painting

(d) Support painting

Figure 6: Dunhuang murals dataset.

Real masks dataset: We extracted real masks from the broken regions of the
mutilated murals are divided into 4 categories according to the breakage rate,
with 1000 images in each category. The real masks can reflect the practical
breakage characteristics of the mutilated murals.

Paris StreetView dataset: A dataset consisting of 15000 images of the main
buildings on the streets of Paris, with 14,900 images in the training set and
100 images in the testing set.

CelebA-HQ dataset: Contains 30,000 high-resolution face images selected from
the CelebA dataset, we randomly selected 29000 images as the training set and
the remaining 1000 as test set.

###  4.3 Comparison models

We compare our method with five representative image inpainting methods in
qualitative and quantitative experiments. The compared advanced methods are:

  * •

CA Yu et al. (2018): An inpainting method using the contextual attention
mechanism established the long-range dependencies of the feature map.

  * •

GC Yu et al. (2019): A coarse-to-fine generative network contains gated
convolution, which is proposed to implement a dynamic feature selection
mechanism.

  * •

LBAM Xie et al. (2019): An end-to-end learnable attention module is designated
to accommodate irregular holes in convolutional layers.

  * •

RFR Li et al. (2020): An image inpainting method depends on recurrent feature
reasoning with consistent attention for large hole regions.

  * •

JPG Guo et al. (2021): A joint predictive method contains a generative network
and filtering with an intelligent combination for murals inpainting.

###  4.4 Qualitative comparisons

Fig. 7 demonstrates the inpainting results using real masks on the Dunhuang
murals dataset. CA and LBAM emerge different degrees of boundary blurring
visually, while GC and JPG suffers from structural distortion and semantic
discontinuities. The similar recurrent inpainting network RFR cannot obtains
smooth inpainting results. In contrast, our inpainting results have a
continuous global structure with precise texture details, which are visually
more realistic. These results show that our MPR module fully uses the
structure and texture information of the murals. In addition, the GFF and MFA
modules effectively realize interaction and extraction of the structure and
texture features, significantly enhancing the inference ability of the model.

Figure 7: Qualitative comparison results of our method with CA, GC, LBAM, RFR
and JPG on the Dunhuang murals dataset using real masks.

###  4.5 Quantitative comparisons

For the evaluation metrics, we use several standard metrics in terms of
picture signal-to-noise ratio peak (PSNR), structural similarity (SSIM), fr
chet inception distance (FID) and learned perceptual image patch similarity
(LPIPS), where the first 2 metrics based on low-level pixel values and the
latter 2 metrics associated with high levels of visual perception related. In
the Dunhuang murals inpainting task, we divided the real mask into 4 groups by
masking rate. For each category, we randomly use 100 masks to test the
performance of our model. As shown in Table 1, our model achieves the best
results for all evaluation metrics on 4 groups of masks.

Table 1: Quantitative comparison results of different mask ratios on Dunhuang murals dataset, $\uparrow$ means higher is better, $\downarrow$ means lower is better, and bold indicates the best score. Metrics | Methods |  (0.01–0.1] |  (0.1–0.2] |  (0.2–0.3] |  (0.3–0.4]
---|---|---|---|---|---
PSNR$\uparrow$ | CA Yu et al. (2018) | 36.04 | 28.55 | 24.67 | 22.39
GConv Yu et al. (2019) | 37.04 | 29.37 | 25.32 | 22.71
LBAM Xie et al. (2019) | 37.01 | 29.61 | 25.83 | 23.76
RFR Li et al. (2020) | 36.46 | 28.83 | 24.92 | 22.45
JPG Guo et al. (2021) | 37.61 | 30.16 | 26.33 | 23.97
OURS | 37.86 | 30.21 | 26.41 | 24.23
SSIM$\uparrow$ | CA Yu et al. (2018) | 0.975 | 0.910 | 0.828 | 0.738
GConv Yu et al. (2019) | 0.979 | 0.921 | 0.843 | 0.749
LBAM Xie et al. (2019) | 0.943 | 0.909 | 0.792 | 0.735
RFR Li et al. (2020) | 0.966 | 0.912 | 0.818 | 0.725
JPG Guo et al. (2021) | 0.985 | 0.938 | 0.844 | 0.759
OURS | 0.986 | 0.942 | 0.853 | 0.785
FID$\downarrow$ | CA Yu et al. (2018) | 10.81 | 31.90 | 64.24 | 83.96
GConv Yu et al. (2019) | 13.82 | 28.54 | 58.48 | 80.77
LBAM Xie et al. (2019) | 13.09 | 36.04 | 63.72 | 81.97
RFR Li et al. (2020) | 13.69 | 41.44 | 62.33 | 81.02
JPG Guo et al. (2021) | 13.13 | 37.69 | 78.48 | 108.07
OURS | 10.45 | 27.84 | 55.59 | 78.86
LPIPS$\downarrow$ | CA Yu et al. (2018) | 0.016 | 0.065 | 0.129 | 0.215
GConv Yu et al. (2019) | 0.013 | 0.051 | 0.109 | 0.189
LBAM Xie et al. (2019) | 0.013 | 0.049 | 0.112 | 0.168
RFR Li et al. (2020) | 0.016 | 0.062 | 0.121 | 0.199
JPG Guo et al. (2021) | 0.015 | 0.067 | 0.139 | 0.227
OURS | 0.011 | 0.044 | 0.104 | 0.166

###  4.6 Experiments on benchmark datasets

We conduct additional experiments to verify the generalization ability of the
model on two benchmark datasets: Paris StreetView and CelebA-HQ, and the
visualization results are shown in Fig. 8. According to the observation, our
proposed method can guarantee the semantic consistency of the restored
content, with smooth and precise edges in the global structure and richer
detail information in the local texture. Table 2 shows the quantitative
comparison results on the Paris StreetView dataset. Our model only slightly
underperforms the LBAM method in LPIPS and FID metrics for small ratio masks
and achieves the best inpainting results in all other comparison metrics.
Overall, our method can maintain excellent inpainting performance in the
quantitative and qualitative comparisons in benchmark datasets.

Figure 8: Qualitative comparison results of our method with CA, GC, LBAM, RFR and JPG on the Paris StreetView and CelebA-HQ datasets using irregular masks. Table 2: Quantitative comparison results of different mask ratios on Paris StreetView dataset, $\uparrow$ means higher is better, $\downarrow$ means lower is better, and bold indicates the best score. Metrics | Methods | (0.01–0.1] | (0.1–0.2] | (0.2–0.3] | (0.3–0.4] | (0.4–0.5] | (0.5–0.6]
---|---|---|---|---|---|---|---
PSNR$\uparrow$ | CA Yu et al. (2018) | 34.35 | 28.55 | 25.67 | 23.39 | 21.22 | 19.18
GConv Yu et al. (2019) | 35.04 | 29.37 | 26.32 | 24.51 | 22.45 | 20.01
LBAM Xie et al. (2019) | 36.11 | 30.57 | 27.39 | 25.23 | 23.30 | 20.91
RFR Li et al. (2020) | 36.36 | 31.17 | 28.08 | 26.01 | 24.15 | 21.29
JPG Guo et al. (2021) | 36.41 | 31.34 | 28.13 | 25.94 | 23.94 | 21.26
OURS | 37.04 | 31.88 | 28.69 | 26.65 | 24.76 | 21.85
SSIM$\uparrow$ | CA Yu et al. (2018) | 0.935 | 0.899 | 0.838 | 0.778 | 0.711 | 0.627
GConv Yu et al. (2019) | 0.959 | 0.921 | 0.853 | 0.813 | 0.741 | 0.645
LBAM Xie et al. (2019) | 0.978 | 0.936 | 0.885 | 0.827 | 0.755 | 0.657
RFR Li et al. (2020) | 0.977 | 0.937 | 0.889 | 0.835 | 0.770 | 0.672
JPG Guo et al. (2021) | 0.978 | 0.939 | 0.892 | 0.836 | 0.776 | 0.692
OURS | 0.986 | 0.953 | 0.894 | 0.853 | 0.791 | 0.699
FID$\downarrow$ | CA Yu et al. (2018) | 12.84 | 27.71 | 52.36 | 64.77 | 80.03 | 121.79
GConv Yu et al. (2019) | 10.82 | 25.54 | 48.98 | 58.77 | 77.79 | 99.91
LBAM Xie et al. (2019) | 7.22 | 20.35 | 35.67 | 51.03 | 69.72 | 96.57
RFR Li et al. (2020) | 9.93 | 23.20 | 42.27 | 56.53 | 82.07 | 108.43
JPG Guo et al. (2021) | 10.86 | 26.52 | 47.81 | 70.72 | 100.73 | 149.32
OURS | 8.93 | 19.73 | 34.34 | 47.18 | 64.32 | 83.25
LPIPS$\downarrow$ | CA Yu et al. (2018) | 0.021 | 0.067 | 0.099 | 0.143 | 0.197 | 0.255
GConv Yu et al. (2019) | 0.018 | 0.052 | 0.089 | 0.129 | 0.175 | 0.268
LBAM Xie et al. (2019) | 0.013 | 0.041 | 0.075 | 0.115 | 0.168 | 0.248
RFR Li et al. (2020) | 0.016 | 0.047 | 0.084 | 0.127 | 0.181 | 0.281
JPG Guo et al. (2021) | 0.022 | 0.065 | 0.119 | 0.179 | 0.229 | 0.304
OURS | 0.015 | 0.042 | 0.075 | 0.112 | 0.161 | 0.242

###  4.7 Ablation studies

In this section, we would like to further validate our contributions. Here we
conduct ablation experiments to research the influences of recurrences in TFR,
and then implement several variants of our model to verify the effectiveness
of the designed modules.

####  4.7.1 The influences of iteration numbers

We fix 4 recurrences in SFR to ensure structural continuity and research the
effect of iteration numbers in TFR on the Dunhuang murals dataset. The results
corresponding to different iteration numbers are shown in Table 3. This
ablation study reveals that our method owns robust, and the improved
performance is from the more efficient architecture rather than the number of
iterations.

Table 3: The influences of different IterNums, the number indicates the iteration times in the second segment. Metrics | IterNum | (0-0.1] | (0.1-0.2] | (0.2-0.3] | (0.3-0.4]
---|---|---|---|---|---
| 3 | 37.56 | 30.02 | 26.19 | 24.01
PSNR$\uparrow$ | 4 | 37.86 | 30.21 | 26.41 | 24.23
| 5 | 37.78 | 30.13 | 26.26 | 24.08
| 3 | 0.963 | 0.924 | 0.841 | 0.773
SSIM$\uparrow$ | 4 | 0.986 | 0.942 | 0.853 | 0.785
| 5 | 0.981 | 0.938 | 0.848 | 0.779

####  4.7.2 The effect of MPR and MFA modules

Previous recurrent inpainting methods tend to a single network architecture,
which is challenging to satisfy the requirements of different receptive fields
for the structure and texture in the mural inpainting tasks. To verify the
superiority of the multi-stage architecture in our model, as the controlled
experiment of a single model, we let the whole recurrent take place in the SFR
under the same recurrences. In addition, we design a second set of control
experiments, which lacks the MFA module compared to the original model. The
qualitative and quantitative comparison results on the Dunhuang murals are
shown in Fig. 9 and Table 4.

Table 4: Quantitative comparison results of the controlled experiments. Metrics | Methods | (0-0.1] | (0.1-0.2] | (0.2-0.3] | (0.3-0.4]
---|---|---|---|---|---
| Single model | 36.77 | 29.38 | 25.89 | 23.73
PSNR$\uparrow$ | Without MFA | 37.62 | 29.97 | 26.29 | 23.81
| Ours | 37.86 | 30.21 | 26.41 | 24.23
| Single model | 0.934 | 0.911 | 0.839 | 0.767
SSIM$\uparrow$ | Without MFA | 0.976 | 0.939 | 0.850 | 0.779
| Ours | 0.986 | 0.942 | 0.853 | 0.785
Figure 9: Qualitative comparison results of controlled experiments.

##  5 Conclusion

In this paper, we propose a multi-stage progressive reasoning network for
murals inpainting containing global to local receptive fields, enriching the
information of missing regions and giving semantically explicit embedding
results. In addition, a multi-scale feature aggregation module is designed to
empower the capability of dynamic selection from significant features. The
effectiveness of the proposed method is proved by conducting experiments on
real damaged murals and comparison with five existing image inpainting
methods. Additionally, we also studied the impact of mural outlines on mural
inpainting tasks. In the future, we aim to further strengthen the constraints
on the contour structure of the murals by introducing an edge extraction
network.

Conflicts of interest The authors declare that they have no conflict of
interest.

Data availability statement Data is openly available in a public repository.

## Acknowledgment

This work was supported in part by the Gansu Provincial Department of
Education University Teachers Innovation Fund Project (No.2023B-056), the
Introduction of Talent Research Project of Northwest Minzu University (No.
xbmuyjrc201904), and the Fundamental Research Funds for the Central
Universities of Northwest Minzu University (No.31920220019, 31920220037,
31920220130), the Leading Talent of National Ethnic Affairs Commission (NEAC),
the Young Talent of NEAC, and the Innovative Research Team of NEAC (2018) 98.

## References

  * Pan and Lu (2003) Pan Y, Lu D (2003) Digital protection and inpainting of dunhuang mural(in chinese). Journal of System Simulation 15(3):310–314
  * Zitová and Flusser (2003) Zitová B, Flusser J (2003) Image registration methods: A survey. Image and Vision Computing 21(11):977–1000
  * Qi et al. (2011) Qi W, Lu D, Zhang H (2011) Virtual completion of facial image in ancient murals. In: Digital Media and Digital Content Management (DMDCM), 2011 Workshop on
  * Criminisi et al. (2003) Criminisi A, Pérez P, Toyama K (2003) Object removal by exemplar-based inpainting. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003), 16-22 June 2003, Madison, WI, USA
  * Tianshui (2011) Tianshui (2011) Dunhuang mural inpainting in intricate disrepaired region based on improvement of priority algorithm. Journal of Computer-Aided Design & Computer Graphics 23(2):284–289
  * Li et al. (2018) Li Q, Wang H, Zou Q (2018) A murals inpainting algorithm based on sparse representation model. Wuhan Daxue Xuebao (Xinxi Kexue Ban)/Geomatics and Information Science of Wuhan University 43(12):1847–1853
  * Liu et al. (2022) Liu B, Du S, Li J, Wang J, Liu W (2022) Dunhuang mural line drawing based on bi-dexined network and adaptive weight learning. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), Springer, pp 279–292
  * Bertalmio et al. (2000) Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp 417–424
  * Ballester et al. (2001) Ballester C, Bertalmio M, Caselles V, Sapiro G, Verdera J (2001) Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing 10(8):1200–1211
  * Tschumperlé and Deriche (2005) Tschumperlé D, Deriche R (2005) Vector-valued image regularization with pdes: A common framework for different applications. IEEE transactions on pattern analysis and machine intelligence 27(4):506–517
  * James et al. (2008) James, Hays, Alexei, A, Efros (2008) Scene completion using millions of photographs. Communications of the ACM 51(10):87–94
  * Sun et al. (2005) Sun J, Yuan L, Jia J, Shum HY (2005) Image completion with structure propagation. In: ACM SIGGRAPH 2005 Papers, pp 861–868
  * Chen et al. (2023) Chen G, Zhang G, Yang Z, Liu W (2023) Multi-scale patch-gan with edge detection for image inpainting. Applied Intelligence 53(4):3917–3932
  * Pathak et al. (2016) Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544
  * Iizuka et al. (2017) Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Transactions on Graphics (ToG) 36(4):1–14
  * Song et al. (2018) Song Y, Yang C, Lin Z, Liu X, Huang Q, Li H, Kuo CCJ (2018) Contextual-based image inpainting: Infer, match, and translate. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 3–19
  * Yan et al. (2018) Yan Z, Li X, Li M, Zuo W, Shan S (2018) Shift-net: Image inpainting via deep feature rearrangement. In: Proceedings of the European conference on computer vision (ECCV), pp 1–17
  * Zeng et al. (2019) Zeng Y, Fu J, Chao H, Guo B (2019) Learning pyramid-context encoder network for high-quality image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1486–1494
  * Huang et al. (2022) Huang Z, Qin C, Weng Z, Zhu Y (2022) Region-aware attention for image inpainting. arXiv preprint arXiv:220401004
  * Yu et al. (2018) Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5505–5514
  * Liu et al. (2019a) Liu H, Jiang B, Xiao Y, Yang C (2019a) Coherent semantic attention for image inpainting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 4170–4179
  * Liu et al. (2019b) Liu G, Shih K, Wang TC, Tao A, Catanzaro B, et al. (2019b) Image in-painting for irregular holes using partial convolutions. US Patent App. 16/360,895
  * Yu et al. (2019) Yu J, Lin Z, Yang J, Shen X, Lu (2019) Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4471–4480
  * Nazeri et al. (2019) Nazeri K, Ng E, Joseph T, Qureshi F, Ebrahimi M (2019) Edgeconnect: Structure guided image inpainting using edge prediction. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)
  * Ren et al. (2019) Ren Y, Yu X, Zhang R, Li TH, Liu S, Li G (2019) Structureflow: Image inpainting via structure-aware appearance flow. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 181–190
  * Guo et al. (2021) Guo X, Yang H, Huang D (2021) Image inpainting via conditional texture and structure dual generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 14134–14143
  * Xiong et al. (2019) Xiong W, Yu J, Lin Z, Yang J, Lu X, Barnes C, Luo J (2019) Foreground-aware image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5840–5848
  * Zhang et al. (2018) Zhang H, Hu Z, Luo C, Zuo W, Wang M (2018) Semantic image inpainting with progressive generative networks. In: Proceedings of the 26th ACM international conference on Multimedia, pp 1939–1947
  * Hochreiter and Schmidhuber (1997) Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural computation 9(8):1735–1780
  * Guo et al. (2019) Guo Z, Chen Z, Yu T, Chen J, Liu S (2019) Progressive image inpainting with full-resolution residual network. In: Proceedings of the 27th acm international conference on multimedia, pp 2496–2504
  * Li et al. (2019) Li J, He F, Zhang L, Du B, Tao D (2019) Progressive reconstruction of visual structure for image inpainting. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 5962–5971
  * Li et al. (2020) Li J, Wang N, Zhang L, Du B, Tao D (2020) Recurrent feature reasoning for image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7760–7768
  * Zeng et al. (2020) Zeng Y, Lin Z, Yang J, Zhang J, Shechtman E, Lu H (2020) High-resolution image inpainting with iterative confidence feedback and guided upsampling. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, Springer, pp 1–17
  * Guo et al. (2019) Guo Z, Chen Z, Yu T, Chen J, Liu S (2019) Progressive image inpainting with full-resolution residual network. ACM
  * Oh et al. (2019) Oh SW, Lee S, Lee JY, Kim SJ (2019) Onion-peel networks for deep video completion. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
  * Quan et al. (2022) Quan W, Zhang R, Zhang Y, Li Z, Wang J, Yan DM (2022) Image inpainting with local and global refinement. IEEE Transactions on Image Processing 31:2405–2420
  * Liu et al. (2018) Liu G, Reda FA, Shih KJ, Wang TC, Tao A, Catanzaro B (2018) Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European conference on computer vision (ECCV), pp 85–100
  * Hu et al. (2018) Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141
  * Xie et al. (2019) Xie C, Liu S, Li C, Cheng MM, Zuo W, Liu X, Wen S, Ding E (2019) Image inpainting with learnable bidirectional attention maps. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 8858–8867
  * Guo et al. (2021) Guo Q, Li X, Juefei-Xu F, Yu H, Liu Y, Wang S (2021) Jpgnet: Joint predictive filtering and generative network for image inpainting. In: Proceedings of the 29th ACM International Conference on Multimedia, pp 386–394