# TriPINet: Tripartite Progressive Integration Network for Image Manipulation
Localization

Wei-Yun Liang, Jing Xu, and Xiao Jin This work was supported in part by the
Science and Technology Planning Project of Tianjin, China (17JCZDJC30700,
18ZXZNGX00310), Tianjin Natural Science Foundation (19JCQNJC00300), and
Fundamental Research Funds for the Central Universities of Nankai University
(63201192, 63211116). (Wei-Yun Liang and Jing Xu contributed equally to this
work.) (Corresponding author: Xiao Jin.) Wei-Yun Liang, Jing Xu and Xiao Jin
are with the College of Artificial Intelligence, Nankai University, Tianjin
300350, China (e-mail: jinxiao@nankai.edu.cn).

###### Abstract

Image manipulation localization aims at distinguishing forged regions from the
whole test image. Although many outstanding prior arts have been proposed for
this task, there are still two issues that need to be further studied: 1) how
to fuse diverse types of features with forgery clues; 2) how to progressively
integrate multistage features for better localization performance. In this
paper, we propose a tripartite progressive integration network (TriPINet) for
end-to-end image manipulation localization. First, we extract both visual
perception information, e.g., RGB input images, and visual imperceptible
features, e.g., frequency and noise traces for forensic feature learning.
Second, we develop a guided cross-modality dual-attention (gCMDA) module to
fuse different types of forged clues. Third, we design a set of progressive
integration squeeze-and-excitation (PI-SE) modules to improve localization
performance by appropriately incorporating multiscale features in the decoder.
Extensive experiments are conducted to compare our method with state-of-the-
art image forensics approaches. The proposed TriPINet obtains competitive
results on several benchmark datasets.

###### Index Terms:

Image forensics, Image manipulation localization, Image tampering detection,
Feature fusion.

##  I Introduction

The proverb “seeing is believing” refers to that what we can visualize, we can
trust and rely on. A foundational assumption of all the image processing
techniques is the authenticity of input images. However, with the rapid
development of image editing tools, digital images can be easily tampered and
spread on websites. Many amateurs without any professional skills can
effortlessly edit images using publicly available tools. Digital image
forgeries inevitably affect our lives. It is harmless to edit images for
entertainment purposes. If image editing is used for military intelligence,
celebrity rumors, fake news on the Internet, insurance defrauding crime,
plagiarism, and other serious issues, it may cause severe consequences. Thus,
the authenticity of image content has always been an important factor for law
enforcement officers to consider. Distinguishing manipulated images from
authentic ones has attracted increasing attention [1, 2, 3].

###  I-A Motivation

####  I-A1 how to fuse different types of manipulation clues

Diverse types of forensic features contribute differently in multimedia
tampering detection [4, 5]. Visual perception information, e.g., RGB inputs,
is insufficient to perform manipulation localization, since visual
imperceptible features, e.g., frequency and noise traces, contain visually
unapparent tampering artifacts [6]. Fusion strategies by concatenation or
elementwise addition operation can not achieve satisfactory performance in
some complex scenes. Thus, how to effectively fuse these features from various
domains becomes a significant problem in this task. Till now, most related
work use early fusion [7, 8, 9] or late fusion [10, 5], only few of them use
multistage middle fusion. In neural networks with multiple branches,
multistage fusion introduces cross-modality interactions into different
layers, which can provide additional guidance for feature learning and
complementarity between low-level and high-level representations [11].
Consequently, to better explore diverse types of tampering traces, a suitable
fusion strategy needs to be further analyzed in this task.

####  I-A2 how to localize the tampered regions in pixel-level

Although multimedia forensics has been studied for years, most prior arts
focus on image-level authenticity classification. Until recently, only few
works analyze tampering localization at pixel-level [12]. In practical
demands, law enforcers should not only classify the fake images, but also
justify where the image areas were tampered [13]. Thus, the localization issue
in image forensics needs further study and improvement. Compared with image-
level classification, pixel-level prediction is a more precise task in which
multiscale features play a significant role. Therefore, how to progressively
integrate the multiscale information in the decoder is of great importance in
pixel-level localization.

###  I-B Contribution

The main contributions of this article can be summed up as follows.

  * •

In this paper, we present a tripartite progressive integration network
(TriPINet), which is an end-to-end triplet architecture designed for image
manipulation localization. Both visual perception information, e.g., RGB
images, and visual imperceptible features, e.g., frequency and noise traces,
are taken into consideration.

  * •

To fuse different types of forensic features, we propose a novel guided cross-
modality dual-attention (gCMDA) module. The feature maps obtained by RGB,
frequency and noise branches are effectively fused by the gCMDA module to
select the most discriminative features for this task.

  * •

We also design progressive integration squeeze-and-excitation (PI-SE) modules
to localize the tampered areas at the pixel-level. Based on the nature of
pooling operations, global max pooling and global average pooling are
alternately adopted in the low-level and high-level stages to progressively
integrate the multiscale features for better localization results.

  * •

Extensive experiments are conducted in several publicly available datasets.
The proposed TriPINet achieves competitive performance against state-of-the-
art forensic approaches under different evaluation metrics. In addition,
ablation studies and robustness analysis are implemented, which demonstrate
the validness of the proposed modules in various conditions.

##  II Related Work

###  II-A Image Manipulation Localization

Most existing image tampering localization methods focus on three semantic
forgeries, including splicing, copy-move and removal [12],[14]. Image splicing
refers to inserting some regions of donor image into another host image [14].
Bi et al. [15] adversarially train two generators based on a multi-decoder-
single-task strategy for image splicing detection and localization. Wang et
al. [16] observe image splicing destroys the periodic correlation pattern
caused by color filter array (CFA) interpolation. They adopt a linear
interpolation model and a covariance matrix to estimate the interpolation
coefficients. Niu et al. [17] break the assumptions in previous splicing
detection based on double compression artifacts. They propose a general
framework to perform image splicing detection, localization and attribution of
tampered regions coming from different donor images.

Copy-move operation duplicates the source regions to the target regions within
the same image. Liu et al. [18] integrate deep matching and keypoint matching
in end-to-end manner. Zhu et al. [19] propose an adaptive attention and
residual refinement network (AR-Net) based on the inconsistence of
illumination and contrast. Islam et al. [20] design a generative adversarial
network (GAN) with a dual-order attention model for image copy-move detection
and localization. In addition, several approaches are proposed for detecting
hybrid types of forgeries [21, 22, 12, 13, 6, 14, 23].

###  II-B Attention Mechanism

The attention mechanism aims to focus on the prominent features in computer
vision tasks. Hu et al. [24] propose a squeeze-and-excitation (SE) block to
flexibly update channel-wise features. Roy et al. [25] combine the channel and
spatial attention units in parallel, called concurrent spatial and channel
squeeze $\&$ excitation (scSE) module. Woo et al. [26] design the
convolutional block attention module (CBAM), which connects channel and
spatial attention in a lightweight manner. Park et al. [27] introduce a
bottleneck attention module (BAM) to select salient areas in a 3D attention
map. Gao et al. [28] develop a global second-order pooling (GSoP) block.

Although attention mechanisms have been utilized for image forensics, some
previous approaches only adopted a single type of attention module . The
effectiveness of the hybrid attention module is not fully explored in this
task. Furthermore, most previous methods employ attention mechanisms to select
discriminative features from only one type of manipulation clues. How to fuse
different types of forensic features with attention modules has received
relatively much less concern. Thus, we develop the gCMDA module to fully
exploit the collaborative relationship among different types of forensic
features. We elaborate this module in Section III-C.

###  II-C Progressive Integration

It is widely acknowledged that the features of different stages are
complementary in the neural networks. The deeper features contain global
information and are more likely to locate the tampered regions, while the
shallower features supply more spatial details. Benefiting from progressively
integrating multiscale features, multistage networks have been widely used in
pixel-level tasks in the computer vision community. Li et al. [29] gradually
integrate multistage feature maps to handle image dehazing in complex scenes.
Mao et al. [30] progressively combine the low-level spatial features with the
high-level semantic features for salient object detection. Chen et al. [31]
observe the complementary information across multiple stages, which is not
fully taken into account by previous works. They present a progressively
complementarity-aware fusion network for RGB-D salient object detection. Liu
et al. [32] leverage multiscale features to produce image manipulation
localization masks in a coarse-to-fine fashion. These methods have proved that
the information in high-level stages can refine the features from low-level
stages.

Thus, we adaptively recalibrate the discriminative features from different
stages. As mentioned in some recent work [33], inappropriate pooling operation
may destroy some valuable information for image manipulation localization.
Therefore, we design the PI-SE module by carefully selecting diverse types of
pooling operations in different stages of the decoder. The details of this
module are presented in Section III-D.

Figure 1: The overall framework of our proposed TriPINet. The encoder extracts
discriminative features from different domains. The gCMDA module is composed
of cross-modality dual attention (CMDA) and guided attention (GA), which is
designed to fuse complementary features. The PI-SE modules are proposed to
progressively integrate multiscale information in decoder. During training,
deep supervision is employed at each stage.

##  III Methodology

###  III-A Overview and Notations

The architecture of the proposed network is shown in Fig. 1. The input of our
network is an RGB image $I\in\mathbb{R}^{H\times W\times 3}$, where $H$ and
$W$ represent the height and the width of the image. The input is first fed
into the encoder to generate features in different domains, e.g., RGB,
frequency and noise domains. We design the gCMDA module, which consists of
cross-modality dual attention and guided attention, to fully explored the
complementary information in different domain features. After obtaining these
hybrid features, we propose the PI-SE modules to integrate multistage
information together for better localization results. During training, deep
supervision is adopted in each stage for fast convergence [34]. The hybrid
loss [35] substitutes for common binary cross-entropy loss [36] to predict
accurate boundary. The descriptions of some notations and operations are
listed in Table I.

TABLE I: Notations and operations used in this paper and their descriptions. Notations | Descriptions
---|---
${\rm Conv}_{k\times k}(\cdot)$ |  $k\times k$ convolutional layers
${\rm Up}_{\times 2}(\cdot)$ | upsampling two times by bilinear interpolation
${\rm BN}(\cdot)$ | batch normalization (BN)
${\rm ReLU}(\cdot)$ | Rectified Linear Unit (ReLU) activation function
${\rm Softmax}(\cdot,\cdot)$ | elementwise Softmax layer
$\sigma(\cdot)$ | Sigmoid activation function
${\rm GAP}(\cdot)$ | global average pooling (GAP)
${\rm GMP}(\cdot)$ | global max pooling (GMP)
${\rm Cat}[\cdot,\cdot]$ | concatenation
${\rm SE}(\cdot)$ | squeeze-and-excitation (SE) attention module
${\rm PISE}_{i}(\cdot)$ | PI-SE attention module in the $i$-th stage of the decoder
$\otimes$ | elementwise multiplication
$\oplus$ | elementwise addition
$i$ | the $i$-th stage in the neural networks
$\mathcal{F}_{i}$ | feature maps outside a module in the $i$-th stage
$f_{i}$ | feature maps inside a module in the $i$-th stage

###  III-B Encoder

The proposed TriPINet is a tripartite network. The encoder of the proposed
network consists of three similar branches, namely the RGB stream, the
frequency stream, and the noise stream, respectively. All the three branches
adopt VGG-16 [37] pre-trained on the ImageNet dataset [38] as backbones to
extract hierarchical features. The output feature maps of each backbone in the
$i$-th stage are denoted as $\mathcal{F}_{i}^{RGB}$, $\mathcal{F}_{i}^{Freq}$
and $\mathcal{F}_{i}^{Noise}$, $i\in\\{1,2,3,4,5\\}$.

####  III-B1 RGB Stream

The RGB stream uses RGB images as inputs. This branch aims to extract visual
perceptible tampering artifacts, such as unnatural tampered region boundaries
and high contrast differences between tampered and authentic regions [6]. If
visual imperceptible features incorrectly suppress some content information,
the semantic clues in RGB images can provide some complementary guidance [5].

####  III-B2 Frequency Stream

Figure 2: The architecture of the DCT extractor in frequency stream. “DCT”
denotes the DCT transformation. “Norm” represents the channel-wise
normalization. “SE” indicates the squeeze-and-excitation module [24].

Frequency domain information may contain key forgery clues. This stream first
adopts a DCT extractor to capture forgery traces [39]. The whole process of
the DCT extractor is illustrated in Fig. 2. Given an RGB image
$I\in\mathbb{R}^{H\times W\times 3}$, it is first converted to the YCbCr color
space and upsampled two times to make the shape compatible to the next phase.
Then, the feature maps are cropped into $N\times N$ patches and discrete
cosine transform (DCT) [40] is performed on these patches to obtain $N\times
N$ DCT coefficients for each channel. Each patch is further cropped into
$n\times n$ blocks. The blocks that represent the same frequency are assigned
in the same channel according to their position in the DCT coefficients.
Channel-wise normalization are further adopted on these feature maps. The
procedure of the reshape operation is shown in Fig. 3.

To deflect irrelevant frequency, we use a squeeze-and-excitation (SE) module
[24] to adaptively recalibrate frequency responses. A shortcut connection is
also integrated into the SE module to enhance the features. After that, the
DCT extractor outputs the frequency features $I^{DCT}\in\mathbb{R}^{H\times
W\times 48}$. The useful frequency information is enhanced and the irrelevant
components are suppressed.

The first stage of the frequency stream uses a $1\times 1$ convolutional layer
and a batch normalization layer to reshape the feature map to the same size of
the second stage. Finally, we obtain the output features of the first stage,
which is denoted as $\mathcal{F}_{1}^{Freq}\in\mathbb{R}^{H\times W\times
64}$. Except for the first stage, the following part of the frequency stream
is the same as the VGG-16 network.

Figure 3: The illustration of the reshape operation in the DCT extractor. The
input feature maps has three channels, and only one channel is shown here for
brevity. Light green represents low frequency features and dark green
signifies high frequency components.

####  III-B3 Noise Stream

To suppress the content information, it is essential to exploit the noise
view. The noise stream consists of a noise extractor and a VGG-16 backbone.
The structure of the noise extractor is shown in Fig. 4. Concretely, the noise
extractor adds the output features of BayarConv layer [41] and SRM layer [42].
The BayarConv layer enhances the noise inconsistency between tampered and
original regions. The trainable kernels of the BayarConv layer have two
constraints in each training iteration, which are defined as,

$\begin{cases}w_{h}(0,0)=-1,\\\ \sum_{m,n\neq 0}w_{h}(x,y)=1,\end{cases}$ (1)

where $w_{h}(x,y)$ denotes the parameter in coordinate $(x,y)$ of the $h$-th
constrained kernel, and $(0,0)$ denotes the center position.

The SRM layer has thirty basic fixed convolutional kernels that extract the
co-occurrence information of the image residuals. It is proved that only three
of the basic SRM kernels have significant effects while applying all thirty
kernels brings little performance improvement [10]. Thus, we only use the
three effective kernels. The detailed parameters of these three kernels are
shown at the bottom of Fig. 4. The kernel size of both BayarConv layer and SRM
layer is $5\times 5\times 3$, and the number of output channels is 3. The
remaining parts of the noise stream are the same as the VGG-16 network.

Figure 4: The architecture of the noise extractor in noise stream.

###  III-C Guided Cross-Modality Dual-Attention (gCMDA) module

After obtaining diverse types of features, we select the most discriminative
parts for further image manipulation localization. Thus, we propose the gCMDA
module. This fusion module consists of two key components, namely cross-
modality dual-attention and guided attention. Fig. 5 illustrates this
structure.

####  III-C1 Cross-Modality Dual-Attention

Parallel spatial attention and channel attention are employed to fuse the the
visual imperceptible features
$\\{\mathcal{F}_{i}^{Freq},\mathcal{F}_{i}^{Noise}\\}\in\mathbb{R}^{H\times
W\times C}$ in the cross-modality dual-attention, which is shown in Fig. 6.

An intermediate branch between spatial and channel attention is added to
enhance the performance. This branch adopts a $1\times 1$ convolutional layer
to reduce the number of channels. The output of this branch is denoted as
$f_{i}^{In}\in\mathbb{R}^{H\times W\times C}$, which can be calculated by the
following formula,

$f_{i}^{In}={\rm Conv}_{1\times 1}({\rm
Cat}[\mathcal{F}_{i}^{Freq},\mathcal{F}_{i}^{Noise}]),$ (2)

where ${\rm Cat}[\cdot,\cdot]$ indicates the concatenation operation, ${\rm
Conv}_{1\times 1}(\cdot)$ represents the $1\times 1$ convolutional layer.

Spatial attention reweights features to focus on some useful spatial
information. In the spatial attention branch, the input features
$\mathcal{F}_{i}^{Freq}$ and $\mathcal{F}_{i}^{Noise}$ are first concatenated,
then the channel number is reduced by two $1\times 1$ convolutional layers.
After that, we use a Softmax layer to obtain the spatial attention weights,
which is denoted as
$\\{\hat{f}_{SA}^{Freq},\hat{f}_{SA}^{Noise}\\}\in\mathbb{R}^{H\times W\times
1}$. This process can be defined as,

$\displaystyle f_{SA}^{Freq}={\rm Conv}_{1\times 1}({\rm Conv}_{1\times
1}({\rm Cat}[\mathcal{F}_{i}^{Freq},\mathcal{F}_{i}^{Noise}])),$ (3)
$\displaystyle f_{SA}^{Noise}={\rm Conv}_{1\times 1}({\rm Conv}_{1\times
1}({\rm Cat}[\mathcal{F}_{i}^{Freq},\mathcal{F}_{i}^{Noise}])),$ (4)
$\displaystyle\\{\hat{f}_{SA}^{Freq},\hat{f}_{SA}^{Noise}\\}={\rm
Softmax}(f_{SA}^{Freq},f_{SA}^{Noise}),$ (5)

where $\\{f_{SA}^{Freq},f_{SA}^{Noise}\\}\in\mathbb{R}^{H\times W\times 1}$
represent intermediate variables. The corresponding elements of
$\hat{f}_{SA}^{Freq}$ and $\hat{f}_{SA}^{Noise}$ sum up to 1. The output of
the spatial attention branch, denoted as $f_{i}^{SA}\in\mathbb{R}^{H\times
W\times C}$, can be expressed as below,

$f_{i}^{SA}=(\hat{f}_{SA}^{Freq}\otimes\mathcal{F}_{i}^{Freq})\oplus(\hat{f}_{SA}^{Noise}\otimes\mathcal{F}_{i}^{Noise})\oplus
f_{i}^{In},$ (6)

where $\otimes$ and $\oplus$ indicate elementwise multiplication and addition,
respectively.

Figure 5: The structure of the gCMDA module. First, the noise and frequency
features are fused by a CMDA module to obtain visual imperceptible features.
Then, the visual imperceptible features and RGB features are further fused by
a GA module to generate the outputs of encoder in each stage.

Channel attention recalibrates channel-wise features to emphasize
discriminative channel feature maps. Compared with the spatial attention
branch, the channel attention branch uses global average pooling (GAP) layers
instead of a $1\times 1$ convolutional layer to obtain intermediate variables
$\\{f_{CA}^{Freq},f_{CA}^{Noise}\\}\in\mathbb{R}^{1\times 1\times C}$. This
procedure can be calculated as follows,

$\displaystyle f_{CA}^{Freq}={\rm GAP}({\rm Conv}_{1\times 1}({\rm
Cat}[\mathcal{F}_{i}^{Freq},\mathcal{F}_{i}^{Noise}])),$ (7) $\displaystyle
f_{CA}^{Noise}={\rm GAP}({\rm Conv}_{1\times 1}({\rm
Cat}[\mathcal{F}_{i}^{Freq},\mathcal{F}_{i}^{Noise}])),$ (8)
$\displaystyle\\{\hat{f}_{CA}^{Freq},\hat{f}_{CA}^{Noise}\\}={\rm
Softmax}(f_{CA}^{Freq},f_{CA}^{Noise}),$ (9)

where ${\rm GAP}(\cdot)$ is the global average pooling layer,
$\\{\hat{f}_{CA}^{Freq},\hat{f}_{CA}^{Noise}\\}\in\mathbb{R}^{1\times 1\times
C}$ represent the channel attention weights. Their corresponding elements sum
up to 1. Assuming $f_{i}^{CA}\in\mathbb{R}^{H\times W\times C}$ indicates the
outputs of the channel attention branch, it can be defined as,

$\displaystyle
f_{i}^{CA}=(\hat{f}_{CA}^{Freq}\otimes\mathcal{F}_{i}^{Freq})\oplus(\hat{f}_{CA}^{Noise}\otimes\mathcal{F}_{i}^{Noise})\oplus
f_{i}^{In}.$ (10)

Finally, the output of the CMDA module, denoted as
$\mathcal{F}_{i}^{CMDA}\in\mathbb{R}^{H\times W\times C}$, is computed as,

$\mathcal{F}_{i}^{CMDA}=f_{i}^{SA}\oplus f_{i}^{CA}.$ (11)

However, visual imperceptible may also suppress some semantic information
incorrectly. In the next subsection, we adopt guided attention to handle this
case.

Figure 6: The structure of the CMDA module. The CMDA consists of three
branches, which are spatial attention branch, intermediante branch and channel
attention branch, respectively.

####  III-C2 Guided Attention

Since the goal of the object-based forgery is to tamper objects in the
multimedia contents, the semantic information in RGB images is also helpful to
localize the tampered areas [5]. It is essential to use complementary
information from hybrid views to generate more discriminative features. In
this paper, we regard RGB features as additional guidance to lead the visual
imperceptible features to focus on the most significant forgery clues.
Inspired by some recent work [43, 44], we name this structure as guided
attention (GA) mechanism in the following. The guided attention is performed
with elementwise multiplication and addition,

$\begin{split}\mathcal{F}_{i}^{En}=(\mathcal{F}_{i}^{RGB}\otimes\mathcal{F}_{i}^{CMDA})\oplus\mathcal{F}_{i}^{RGB},\end{split}$
(12)

where $\mathcal{F}_{i}^{En}$ denotes the output of the guided attention. In
this way, the additional RGB features $\mathcal{F}_{i}^{RGB}$ complements the
semantic information that may be incorrectly suppressed by visual
imperceptible features $\mathcal{F}_{i}^{CMDA}$ [43]. $\mathcal{F}_{i}^{En}$
is also the output of the encoder in the $i$-th stage. In the next subsection,
$\mathcal{F}_{i}^{En}$ will be fed into the decoder.

###  III-D Progressive Integration SE (PI-SE) module

Global average pooling (GAP) and global max pooling (GMP) are commonly used in
neural networks. GAP and GMP play different roles in the pooling operation.
GAP generates the averaged features of the inputs, while GMP captures the most
significant values. As a result, GAP preserves the complete region of the
object, whereas GMP pay more attention to the prominent information in the
feature maps and might miss out in some detail [30]. It is important to take
advantages of these two pooling strategies in attention modules. To recover
the feature maps in the decoder, we develop a Progressive Integration SE (PI-
SE) module, which uses distinct attention structures in different stages.

According to the data flow direction in decoder, we describe the PI-SE from
high-level to low-level. In the third to the fifth stages of the decoder, the
resolution of the high-level feature maps is relatively low. Thus, we use PI-
SE with GAP to recover the complete manipulated regions. Assuming the input of
the PI-SE module is denoted as $\mathcal{F}_{i}^{Fus}$, this calculation
process can be expressed by,

$\begin{split}f_{i}^{PISE}=&\sigma({\rm Conv}_{1\times 1}({\rm ReLU}({\rm
Conv}_{1\times 1}({\rm GAP}(\mathcal{F}_{i}^{Fus}))))),\\\
\mathcal{F}_{i}^{PISE}=&(f_{i}^{PISE}\otimes\mathcal{F}_{i}^{Fus})\oplus\mathcal{F}_{i}^{Fus},i\in\\{3,4,5\\},\end{split}$
(13)

where $f_{i}^{PISE}\in\mathbb{R}^{1\times 1\times C}$ indicates the attention
weights in the PI-SE module and $\mathcal{F}_{i}^{PISE}\in\mathbb{R}^{H\times
W\times C}$ denotes the output features of the PI-SE module. Fig. 7(a)
illustrates this case.

In the first two stages of the decoder, the tampering regions have been
completely restored by the PI-SE in high-level stages. Therefore, PI-SE with
GMP is adopted to focus on the important information and ignore false alarm
areas. Meanwhile, the boundaries of the predicted feature maps can be refined
in detail [30]. The PI-SE with GMP can be calculated by,

$\begin{split}f_{i}^{PISE}=&\sigma({\rm Conv}_{1\times 1}({\rm ReLU}({\rm
Conv}_{1\times 1}({\rm GMP}(\mathcal{F}_{i}^{Fus}))))),\\\
\mathcal{F}_{i}^{PISE}=&(f_{i}^{PISE}\otimes\mathcal{F}_{i}^{Fus})\oplus\mathcal{F}_{i}^{Fus},i\in\\{1,2\\},\end{split}$
(14)

where ${\rm GMP}(\cdot)$ stands for global max pooling. Fig. 7(b) describes
this process intuitively.

(a) PI-SE module in stage 3, 4, 5.

(b) PI-SE module in stage 1, 2.

Figure 7: The illustration of the PI-SE modules in different stages. According
to the data flow direction in decoder, we use PI-SE module with GAP in the
3rd-5th stages, and PI-SE modules with GMP are adopt in the 1st-2nd stages.

###  III-E Decoder

Decoder aims to gradually increase the resolutions of the feature maps to the
raw size of the input images. The decoder is divided into five stages. Each
stage contains a PI-SE module followed by a series of operations. The input of
each PI-SE module in the $i$-th stage involves multiscale information, which
is denoted as $\mathcal{F}_{i}^{Fus}$,

$\displaystyle\mathcal{F}_{i}^{Fus}=\left\\{{\begin{array}[]{*{20}{l}}{\mathcal{F}_{i}^{En}\oplus\mathcal{F}_{i+1}^{De},i\in\\{1,2,3,4\\},}\\\
{\mathcal{F}_{i}^{En},i=5,}\end{array}}\right.$ (17)

where $\mathcal{F}_{i}^{En}$ is the output of the encoder in the $i$-th stage,
$\mathcal{F}_{i}^{De}$ is the output of the $i$-th block in the decoder.

For brevity, we use “Block” to represent a set of operations in Fig. 1. Each
block is composed of a $1\times 1$ convolutional layer, a batch normalization
(BN) layer, a rectified linear unit (ReLU) activation function, and bilinear
interpolation for upsampling. The main purpose of these operations is to make
the shape compatiable among different stages. Then, the output of each stage
in decoder can be formulated as,

$\displaystyle\mathcal{F}_{i}^{De}={\rm Up_{\times 2}(ReLU(BN}({\rm
Conv}_{1\times 1}({\rm PISE}_{i}(\mathcal{F}_{i}^{Fus}))))),$ (18)

where ${\rm PISE}_{i}(\cdot)$ denotes the PI-SE module we described in the
above subsection.

After obtaining $\mathcal{F}_{1}^{De}$, we add a $1\times 1$ convolutional
layer to reduce the channel number to 1. The size of the final feature map
output by the decoder is the same as the input images.

###  III-F Loss Function

The proportion of tampered pixels in a manipulated image is relatively small,
which leads to a class imbalance problem. Since a large number of negative
pixels contribute little to the learning process, the imbalance problem will
make the model tends to classify the pixels as authentic [45, 46]. The popular
binary cross-entropy (BCE) loss [47] treats the negative and positive samples
equally. Thus, the important information in the positive pixels cannot be
fully explored.

To address this issue, we adopt a hybrid loss that focus on the positive
pixels in both tampered boundaries and manipulated regions [19][35][48]. The
hybrid loss consists of the BCE loss, the intersection over union (IoU) loss
[49] and the structural similarity (SSIM) loss [50],

$\begin{split}\mathcal{L}_{k}=\mathcal{L}_{k}^{BCE}+\mathcal{L}_{k}^{SSIM}+\mathcal{L}_{k}^{IoU},\end{split}$
(19)

where $k$ stands for the indexes of the outputs in the deep supervision [34].

The binary cross-entropy (BCE) treats the foreground and background pixels
equally. The BCE loss is defined as,

$\mathcal{L}^{BCE}=-\sum\limits_{j}{[{G_{j}}\log({P_{j}})+(1-{G_{j}})\log(1-{P_{j}})]},$
(20)

where $P$ is the predicted feature maps, $G$ is the groundtruth and $j$
denotes the pixel coordinate.

The SSIM loss measures the structural similarity between prediction and
groundtruth in patch-level. It gives higher weights to boundary pixels that
locate between tampered and authentic regions. As a result, the structural
information in the boundary areas is retained. The SSIM loss can be expressed
by,

$\mathcal{L}^{SSIM}=1-\frac{{(2{\mu_{P}}{\mu_{G}}+{\epsilon_{1}})(2{\sigma_{PG}}+{\epsilon_{2}})}}{{(\mu_{P}^{2}+\mu_{G}^{2}+{\epsilon_{1}})(\sigma_{P}^{2}+\sigma_{G}^{2}+{\epsilon_{2}})}},$
(21)

where $\mu_{P},\mu_{G}$ and $\sigma_{P},\sigma_{G}$ are the means and the
standard deviations of two patches from the prediction $P$ and the groundtruth
$G$, respectively. $\sigma_{PG}$ is the covariance matrix.
${\epsilon_{1}}=0.01^{2}$ and ${\epsilon_{2}}=0.03^{2}$ are constant numbers
to prevent division by zero.

The IoU loss, which is often used in object detection and segmentation,
emphasizes more on the large foreground regions. It is formulated as,

$\mathcal{L}^{IoU}=1-\sum\limits_{j}{\frac{{{P_{j}}{G_{j}}}}{{{P_{j}}+{G_{j}}-{P_{j}}{G_{j}}}}}.$
(22)

The outputs of the decoder are used for the final manipulation localization.
Besides the final outputs, we also add deep supervision [34] in each stage of
the decoder. In this way, our network is supervised with six outputs. Finally,
the training loss is defined as the summation of the hybird loss for all the
outputs,

$\mathcal{L}=\sum_{k=1}^{6}\mathcal{L}_{k},$ (23)

where $k\in\\{1,2,3,4,5,6\\}$ denotes the $k$-th supervised output. In the
decoder, the predicted feature maps will be upsampled by bilinear
interpolations to make the resolution consistent with the groundtruth.

##  IV Experiments

###  IV-A Experimental Setup

####  IV-A1 Datasets

We conduct experiments on seven public benchmark datasets to verify the
effectiveness of our proposed method. CASIA [51] is composed of two versions.
CASIAv1 has 461 splicing and 459 copy-move images, and the tampered images are
not post-processed. CASIAv2 has 1,828 splicing, 3,295 copy-move and 7,491
authentic images. Post-processing operations such as filtering and blurring
are applied to the tampered images. COLUMB [52] contains 184 authentic and 180
splicing images. The source tampered images are not post-processed. Carvalho
[53] has 100 splicing and 100 authentic images. The manipulated contents are
all people. It is post-processed by adjusting contrast and illumination.

The above three datasets are used for splicing localization, while the
following four datasets are adopted for copy-move localization. Synthetic
Dataset [23] includes 6,200 image copy-move examples, which is subjected to
resizing and rotation in different factors. CASIA-CMFD is produced by [54]. It
is a subset of the CASIAv2 dataset. There are 1,313 copy-move images and their
authentic counterparts. CoMoFoD [55] contains 200 source copy-move images.
Each source tampered image will undergo 25 post-processing/attacks to produce
another 25 tampered images. COVERAGE [56] is a challenge copy-move dataset,
which contains 100 copy-move images and their corresponding ground-truth
masks.

####  IV-A2 Evaluation Metrics

Image manipulation localization is a pixel-level binary classification
problem, which infers whether each pixel is tampered or not. Following
previous works [45, 54], we use F1-score and Intersection over Union (IoU) to
quantify the performance. These metrics can be calculated by,

$F1=\frac{2TP}{2TP+FN+FP},$ (24) $IoU=\frac{TP}{TP+FN+FP},$ (25)

where $TP$ represents the number of correctly classified manipulated pixels;
$FN$ denotes the number of misclassified tampered pixels; $TN$ indicates the
number of correctly classified authentic pixels; $FP$ is the number of wrongly
classified original pixels.

TABLE II: Data split protocol for image splicing and copy-move localization, respectively. Dataset | Train | Validation | Test
---|---|---|---
CASIAv2 [51] | 4500 | 122 | 501
CASIAv1 [51] | 341 | 20 | 100
COLUMB [52] | 125 | 5 | 50
Carvalho [53] | 70 | 10 | 20
Synthetic Dataset [23] | 4960 | 620 | 620
CASIA-CMFD [54] | 1051 | 131 | 131
CoMoFoD [55] | 180 | 20 | 20
COVERAGE [56] | 80 | 10 | 10

####  IV-A3 Comparative Methods

Since the performance of the methods based on deep learning has outperforms
the conventional approaches by a large margin, we only use the frameworks
based on deep learning for comparison. In image splicing localization, 11
state-of-the-art frameworks are included.

  * •

DenseFCN [13] is a fully convolutional network, which adopts dense connections
and dilated convolutions for better localization.

  * •

MFCN [57] is a multitask dual-stream network, which mines traces in
manipulated region and forged boundaries, respectively.

  * •

J-LSTM [58] combines LSTM with CNN to learn the boundary discrepancy.

  * •

H-LSTM [23] introduces resampling features in LSTM to discover the transition
between tampered and authentic patches.

  * •

SG-CRF [59] leverages local and global features together to obtain better
localization performance.

  * •

ManTra-Net [7] formulates the tampering localization problem as a local
anomaly detection problem. It adopts self-supervised learning to learn robust
tampered traces.

  * •

RRU-Net [60] is a ringed residual U-Net for image forgery detection.

  * •

MTSE-Net [45] incorporates multitask learning with squeeze-and-excitation
attention for splicing localization.

  * •

C2R-Net [61] cascades a coarse convolutional neural netowrk and a refined CNN.

  * •

RTAG [15] uses a generative adversarial network framework for splicing
localization.

  * •

MCNL-Net [62] employs U-Net controlled by multiple scales to localize the
tampered regions.

In copy-move localization, two popular methods are involved for comparison.

  * •

BusterNet [54] uses the Simi-Det branch to detect copy-move regions. Following
[19, 63, 18], we compare with the similar branch of
BusterNet.111https://github.com/ntdat017/BusterNet_pytorch

  * •

CMSDNet [63] adopts double-level self-correlation and dilated convolutions to
extract high-resolution features.222https://github.com/imagecbj/A-serial-
image-copy-move-forgery-localization-scheme-with-source-target-distinguishment

###  IV-B Training and Implementation Details

Our models are implemented in PyTorch and trained on an NVIDIA RTX 3080Ti GPU.
All the input images are resized to $256\times 256$. We perform data
augmentation, such as random horizontal and vertical flipping, on the training
data. The VGG-16 backbones are initialized with the weights that are pre-
trained on the ImageNet dataset [38]. The proposed network is trained end-to-
end using the Adam optimizer with a weight decay of 5e-4. The learning rate is
initialize to 1e-3 and reduced to half if the validation loss does not drop
for five epochs. The batch size is set as 16. We train our model for 100
epochs. The model that achieves the best performance on validation set is
chose as the final model to evaluate the performance on test set. Following
the same protocol in [10, 57, 23], we vary different thresholds and use the
highest metric score as the final score.

####  IV-B1 Image Splicing Localization

In this task, we follow the exact same experimental protocol defined in [45].
We randomly split the data in CASIAv2, CASIAv1, COLUMB and Carvalho datasets
for evaluation. The details of the split ratios are shown in Table. II. To
address the overfitting caused by data deficiency, we pre-trained our model on
CASIAv2. Then, we further fine-tuned our model on the other three small
datasets. Note that fine-tuning is a widely-used protocol in image splicing
localization [10, 8].

####  IV-B2 Image Copy-Move Localization

In image copy-move localization, the source and target pixels are both labeled
as forged [54, 19, 63, 18]. Previous methods often use a large self-made
dataset for pre-training, and evaluate without fine-tuning. Since these self-
made datasets are either private or no longer publicly accessible, we use a
subset of the Synthetic Dataset [23] for pre-
training.333https://www.dropbox.com/sh/palus3sq4zvdky0/AACu3s7KA5Fhr_
BJUeDOxnTLa?dl=0 In the pre-training phase, we randomly divided the Synthetic
Dataset into the training, validation and test sets according to the ratio of
8:1:1, which is suggested in [54]. Then, the pre-trained models are evaluated
on CASIA-CMFD, CoMoFoD, COVERAGE, and remaining samples in the Synthetic
Dataset to verify the generalization ability. Additionally, we also provide
performance comparison after fine-tuning on CASIA-CMFD, CoMoFoD, and COVERAGE
datasets. The dataset partition is shown in Table. II. Fine-tuning is a common
protocol in image splicing localization, while it is neglected by some
previous copy-move detection methods.

TABLE III: Performance on pixel-level image splicing localization. Best result per test set is highlighted in bold. All methods are evaluated following the same protocol defined in [45] except for those marked with asterisks (*).  Method | CASIAv2 [51] |  | CASIAv1 [51] |  | COLUMB [52] |  | Carvalho [53] |  | Average
---|---|---|---|---|---|---|---|---|---
F1 | IOU |  | F1 | IOU |  | F1 | IOU |  | F1 | IOU |  | F1 | IOU
J-LSTM (2017ICCV) [58] | 0.167 | 0.106 |  | 0.431 | 0.309 |  | 0.275 | 0.154 |  | 0.278 | 0.169 |  | 0.288 | 0.185
SG-CRF (2018ECCV) [59] | 0.145 | 0.091 |  | 0.166 | 0.098 |  | 0.256 | 0.172 |  | 0.234 | 0.136 |  | 0.200 | 0.124
MFCN (2018JVCIR) [57] | 0.376 | 0.304 |  | 0.423 | 0.340 |  | 0.622 | 0.529 |  | 0.321 | 0.215 |  | 0.436 | 0.347
H-LSTM (2019TIP) [23] | 0.194 | 0.134 |  | 0.269 | 0.186 |  | 0.270 | 0.175 |  | 0.161 | 0.093 |  | 0.224 | 0.147
ManTra-Net (2019CVPR) [7] | 0.201 | 0.126 |  | 0.226 | 0.139 |  | 0.472 | 0.328 |  | 0.325 | 0.202 |  | 0.306 | 0.199
RRU-Net (2019CVPRW) [60] | 0.533 | 0.475 |  | 0.481 | 0.426 |  | 0.731 | 0.669 |  | 0.247 | 0.180 |  | 0.498 | 0.438
DenseFCN (2021TIFS) [13] | 0.098 | 0.069 |  | 0.174 | 0.125 |  | 0.323 | 0.231 |  | 0.271 | 0.164 |  | 0.217 | 0.147
MTSE-Net (2022TCSVT) [45] | 0.765 | 0.714 |  | 0.667 | 0.592 |  | 0.910 | 0.867 |  | 0.353 | 0.251 |  | 0.674 | 0.606
C2R-Net* (2020IS) [61] | 0.676 | - |  | - | - |  | 0.695 | - |  | - | - |  | - | -
MCNL-Net* (2020TOMM) [62] | 0.866 | - |  | - | - |  | 0.772 | - |  | - | - |  | - | -
RTAG* (2021ICCV) [15] | 0.815 | - |  | - | - |  | 0.823 | - |  | - | - |  | - | -
Ours+VGG11 | 0.862 | 0.758 |  | 0.809 | 0.679 |  | 0.948 | 0.901 |  | 0.551 | 0.380 |  | 0.793 | 0.680
Ours+VGG16 | 0.904 | 0.824 |  | 0.812 | 0.683 |  | 0.962 | 0.927 |  | 0.566 | 0.395 |  | 0.811 | 0.707
Figure 8: Splicing localization results of different methods (F1-score/IoU).

###  IV-C Comparison on Image Splicing Localization

Quantitive comparison on splicing localization is listed in Table. III. Since
we follow the exactly same protocol with [45], the evaluation results of other
methods in Table III are reported from [45]. There are three other approaches
using the same datasets with different data split ratios [61, 62, 15]. We mark
these three with an asterisk (*). Compared the splicing localization
performance F1-score/IoU with MTSE-Net, it can be observed that our method is
13.9%/11.0% higher on CASIAv2, 14.5%/9.1% higher on CASIAv1, 5.2%/6.0% higher
on COLUMB and 21.3%/14.4% higher on Carvalho. The “Average” represents the
average performance over the four standard datsets. The proposed TriPINet
achieves an average F1-score/IoU score of 81.1%/70.7%, which is 13.7%/10.1%
higher than the second-best method, e.g., MTSE-Net [45]. Our model achieves
competitive performance against other existing splicing localization methods
on the four standard datasets. The reason can be explained as follows.
Different from other methods only use simple fusion strategies, we design the
gCMDA module to fully explore the complementary information in visual
perceptible and imperceptible features. In addition, we introduce PI-SE
modules to gradually refine the feature maps by multiscale information.

More intuitively, Fig. 8 shows the qualitative localization results. The
localization maps of J-LSTM [58] and SG-CRF [59] shows apparent block
artifacts because both of these two methods train images in a patch manner. It
is also difficult for other mentioned approaches to detect complete outlines
of the manipulated regions. MTSE-Net [45] obtains relatively good performance.
However, it still wrongly localize extra objects as spliced regions, as shown
in the second and the third rows of Fig. 8. In the first three rows, the
proposed TriPINet can precisely localize the tampered regions. In the fourth
example, although the quantitative metrics of our proposed method are slightly
lower than MTSE-Net, the qualitative results are still comparable.

TABLE IV: Performance of pixel-level image copy-move localization of different pre-trained models. The models are trained on Synthetic Dataset [23]. Best result per test set is highlighted in bold. Method | Synthetic [23] |  | CASIA-CMFD [54] |  | CoMoFoD [55] |  | COVERAGE [56] |  | Average
---|---|---|---|---|---|---|---|---|---
F1 | IoU |  | F1 | IoU |  | F1 | IoU |  | F1 | IoU |  | F1 | IoU
BusterNet (2018ECCV) [54] | 0.872 | 0.772 |  | 0.240 | 0.136 |  | 0.189 | 0.105 |  | 0.510 | 0.342 |  | 0.453 | 0.339
CMSDNet (2021TMM) [63] | 0.932 | 0.872 |  | 0.285 | 0.166 |  | 0.215 | 0.120 |  | 0.565 | 0.393 |  | 0.499 | 0.388
Ours | 0.963 | 0.929 |  | 0.276 | 0.160 |  | 0.246 | 0.140 |  | 0.622 | 0.451 |  | 0.527 | 0.420
TABLE V: Performance of pixel-level image copy-move localization using fine-tuned models. The models are pre-trained on Synthetic Dataset [23] and fine-tuned on each dataset. Best result per test set is highlighted in bold. Method | CASIA-CMFD [54] |  | CoMoFoD [55] |  | COVERAGE [56] |  | Average
---|---|---|---|---|---|---|---
F1 | IoU |  | F1 | IoU |  | F1 | IoU |  | F1 | IoU
BusterNet (2018ECCV) [54] | 0.371 | 0.228 |  | 0.823 | 0.700 |  | 0.677 | 0.512 |  | 0.624 | 0.480
CMSDNet (2021TMM) [63] | 0.446 | 0.287 |  | 0.904 | 0.825 |  | 0.706 | 0.546 |  | 0.685 | 0.553
Ours | 0.472 | 0.309 |  | 0.958 | 0.919 |  | 0.751 | 0.602 |  | 0.727 | 0.610
Figure 9: Copy-move localization results of different methods (F1-score/IoU).

###  IV-D Comparison on Image Copy-Move Localization

In this subsection, we first pre-trained the proposed TriPINet and other
compared methods on the Synthetic Dataset [23]. Then, these pre-trained models
are tested on CASIA-CMFD, CoMoFoD, COVERAGE, and remaining samples in the
Synthetic Dataset. As shown in Table. IV, our method outperforms other
approaches on Synthetic Dataset [23], CoMoFoD and COVERAGE. On the CASIA-CMFD
dataset, the F1-score/IoU of our method is slightly lower than CMSDNet by
0.9%/0.6%. We try to explain the reason for these observations as follows. The
CASIA-CMFD dataset is a challenging dataset, whose data distribution is very
different from Synthetic Dataset [23]. Additionally, the copy-move detection
schemes usually adopt self-correlation modules to match patch-like features
[54, 19, 63, 18]. Without such a highly customized module for copy-move
localization, our performance will inevitably degrade. However, the proposed
TriPINet still achieves comparable results on the average of the four
datasets.

After that, we fine-tuned these pre-trained models on CASIA-CMFD, CoMoFoD, and
COVERAGE. As shown in Table. V, after fine-tuning, our method outperforms
existing copy-move localization methods on these three datasets. Although not
equipped with self-correlation modules, our model can still achieve
competitive performance for copy-move localization. Usually, the pre-training
datasets for copy-move detection contain one hundred thousand samples [54].
The annotation and training of such a large dataset are labor and time
consuming. The Synthetic Dataset we used only has 6,200 samples, nearly 6% of
some large datasets. Fine-tuning can boost performance even without large-
scale pre-training, which speeds up the training process.

The qualitative results of copy-move localization methods are shown in Fig. 9.
In these examples, our model provides more accurate results. For instance, in
the second row, BusterNet and CMSDNet can coarsely locate the forged regions
without precise contours. The third sample has multiple copy-move objects.
BusterNet and CMSDNet can not identify all the tampered objects in this
challenging case. In contrast, our method not only locates all the manipulated
areas, but also obtains sharp boundaries. In the last row of Fig. 9, only our
method generates satisfactory localization results.

###  IV-E Ablation Study

In this subsection, we testify the necessity of some key components in the
proposed TriPINet. In each ablation experiment, only one component is
modified, while others remain unchanged. Then, we retrain the neural networks
on CASIAv2 dataset with the same training protocol as before and report the
results.

####  IV-E1 The Effectiveness of Different Backbones

TABLE VI: Ablation studies for backbones. These backbones are initialized with the weights pre-trained on the ImageNet dataset. Then, our model with different backbones are trained and tested on CASIAv2 dataset. Backbone | CASIAv2
---|---
F1 | IoU
ResNet-18 | 0.717 | 0.559
ResNet-34 | 0.737 | 0.584
ResNet-50 | 0.788 | 0.650
VGG-11 | 0.862 | 0.758
VGG-16 | 0.904 | 0.824
VGG-19 | 0.807 | 0.677

In this subsection, we evaluate our model with different backbones. These
backbones are initialized with the weights that are pre-trained on the
ImageNet dataset [38]. As shown in Table. VI, the performance of VGG networks
are generally better than ResNet backbones. Since the VGG model is shallower
than ResNet, the low-level features can supply more discriminative features
for image manipulation localization [45]. Our method achieves the best
performance when using VGG-16 as backbones.

####  IV-E2 The Significance of the gCMDA module

TABLE VII: Ablation studies for the gCMDA module. “Freq-Noise” column denotes the fusion method for visual imperceptible featues. “RGB” column indicates the operation for fusing RGB features with the visual imperceptible features. Feature Fusion |  | CASIAv2
---|---|---
Freq-Noise | RGB |  | F1 | IoU
Add | Add |  | 0.784 | 0.644
Cat | Cat |  | 0.705 | 0.545
Add | GA |  | 0.787 | 0.649
Cat | GA |  | 0.817 | 0.690
CMDA | Add |  | 0.800 | 0.667
CMDA | Cat |  | 0.710 | 0.550
CMDA | GA |  | 0.862 | 0.758

As shown in Table. VII, we conduct experiments to demonstrate the
effectiveness of both CMDA and GA. In the first column of Table. VII, “Freq-
Noise” stands for how to fuse the features from frequency stream and noise
stream. “RGB” in the second column represents how to fuse visual imperceptible
features with RGB information. “Add” denotes elementwise addition, and “Cat”
indicates concatenate operation. It is obvious that the proposed gCMDA module
can extract more discriminative cross-modality features for tampering
localization. The reason can be interpreted as below. The dual attention
structure in CMDA module emphasizes the important features, while suppressing
the redundant information both in channel and spatial dimensions. Moreover,
the GA module uses RGB features as complementary semantic information. Since
image manipulation aims to change the object layouts in the images, semantic
clues is also useful in this task [5]. In the second column of Fig. 10, we
illustrate some localization examples without gCMDA modules. Intuitively, our
model can not obtain satisfactory results without the gCMDA module. In this
circumstance, hybrid features are not effectively fused.

####  IV-E3 The Effectiveness of the PI-SE

TABLE VIII: Ablation studies for the PI-SE module. “3GAP+2GMP” indicates the decoder consists of three PI-SE modules with GAP in high-level stages, and two PI-SE modules with GMP in low-level stages. PI-SE | CASIAv2
---|---
F1 | IOU
5GAP (w/o PI-SE) | 0.797 | 0.663
4GAP+1GMP | 0.816 | 0.689
3GAP+2GMP | 0.862 | 0.758
2GAP+3GMP | 0.821 | 0.698
1GAP+4GMP | 0.797 | 0.676
5GMP | 0.800 | 0.682
4GMP+1GAP | 0.816 | 0.698
3GMP+2GAP | 0.774 | 0.631
2GMP+3GAP | 0.789 | 0.651
1GMP+4GAP | 0.799 | 0.665

In pixel-level visual tasks, how to gradually increase the feature maps in the
decoder is vital for the final results [29, 30, 31]. To validate the proposed
PI-SE modules, we alternately change the pooling operations, drawing the
following observations. (1) In the first row of Table. VIII, “5GAP (w/o PI-
SE)” denotes that all the PI-SE modules are replaced by the SE module [24].
Compared with conventional SE modules, hybrid pooling strategies bring obvious
performance improvements in most cases. The advantage of GAP and GMP are fully
explored by PI-SE modules in different stages. (2) GAP at high-level stages
and GMP at low-level stages perform better than the opposite order. In the
high-level stages, each pixel is corresponding to a large receptive field,
which contains significant information. We use PI-SE with GAP to recover the
manipulated regions. In the low-level stages, the feature maps are restored by
previous stages in the decoder. Thus, PI-SE with GMP is adopted to focus on
the prominent information, ignoring abundant features such as false alarm
areas. As a result, the boundaries of the predicted feature maps can be
refined in detail [30]. In the fourth column of Fig. 10, “w/o PI-SE”
represents for replacing PI-SE with conventional SE module. The proposed PI-SE
module is helpful to precisely recover the feature maps.

####  IV-E4 The Superiority of the Hybrid Loss

TABLE IX: Ablation studies for the hybrid loss function. Loss | CASIAv2
---|---
F1 | IOU
BCE | 0.640 | 0.470
BCE+SSIM | 0.631 | 0.461
BCE+IOU | 0.843 | 0.729
BCE+IOU+SSIM | 0.862 | 0.758

In Table. IX, we investigate the influence of hybrid loss. Comparing with
simple BCE loss, the hybrid loss obtains a improvement by 22.2%/25.9% in terms
of F1-score/IoU, which demonstrate the superiority of the hybrid loss. In the
second column of Fig. 10, the results supervised with BCE loss can only
provide coarse shape of the manipulated regions. This demonstrates the hybrid
loss can supervise the training process more effectively.

Figure 10: Ablation analysis of the proposed TriPINet (F1-score/IoU). “only
BCE” denotes the model is only supervised by BCE loss. “w/o gCMDA” indicates
the model uses simple addition operations to replace the gCMDA module. “w/o
PI-SE” means that all the PI-SE modules are substituted with the SE module
[24].

###  IV-F Robustness Analysis

(a) Against JPEG compression.

(b) Against Gaussian Blurs.

Figure 11: Robustness evaluation against two image post-processing operations,
e.g., JPEG Compression and Gaussian Blur.

We analyze the robustness of TriPINet against two common post-processing
techniques when images transmitted on the Internet [10, 7, 8, 5], i.e. JPEG
compression and Gaussian blur. Following the robustness evaluation protocol
defined in [5], CASIAv2 is used for training, and CASIAv1 is adopted for test.
Both the splicing and the copy-move forgeries are included. H-LSTM [23] and
ManTra-Net [7] are employed for comparison.

####  IV-F1 Robustness Against JPEG Compression

In the experiments, JPEG quality factors are ranged from 50 to 100, with the
step of 10. Fig. 11(a) demonstrates that our method obtains the best
localization results among the mentioned methods. When JPEG quality factor is
below 70, image visual degradation and blocking artifact are noticeably
visible [55]. Our method can still correctly locate some forgeries in this
challenging case.

####  IV-F2 Robustness Against Gaussian Blurs

In this subsection, we use five sizes of blur kernels, including $5\times 5$,
$11\times 11$, $17\times 17$, $23\times 23$ and $29\times 29$. In Fig. 11(b),
the proposed TriPINet obtains the best localization results among the compared
methods. As the kernel size increased, the F1-score of TriPINet drops. The
reason is inferred that the statistic distribution of blurred images is
obviously different from the original ones, especially when the kernel size is
bigger than $7\times 7$ [55].

###  IV-G Failure Cases and Analysis

Figure 12: Some failure cases of our proposed TriPINet.

It is worthwhile to study the reasons for failure cases. The unsatisfactory
results are mainly caused by three circumstances. First, the tampered regions
are the background, occupying large areas. Our method will wrongly distinguish
foreground and background in such situation, as demonstrated in the first row
of Fig. 12. Second, when the foreground is small and the background contains
complex scenes, the proposed TriPINet can not recognize tampered pixels well,
such as the second row in Fig. 12. Finally, if the tampered object is complex
with fine structures, our framework can not generate all the specifics of
manipulated areas. This case is illustrated in the third row of Fig. 12. The
tampered object is a tree with intricate details, such as lots of thin
branches and small fruits.

##  V Conclusion

In this paper, we propose a tripartite progressive integration network
(TriPINet) for end-to-end image tampering localization. The gCMDA module is
designed to fuse the complementary information both in visual perceptible and
visual imperceptible domains. The PI-SE module is developed in the decoder
stages to appropriately incorporate multiscale features and gradually recover
the feature maps in detail. Our TriPINet is compared with 13 state-of-the-art
methods on seven benchmark datasets, achieving comparable performance both in
splicing and copy-move localization tasks. Ablation studies demonstrate the
effectiveness of the proposed modules. Experimental results under attacks,
e.g., JPEG compression and Gaussian blur, also show the robustness of the
proposed TriPINet. Finally, some failure cases are discussed for further
improvements in the future.

## References

  * [1] L. Verdoliva, “Media Forensics and DeepFakes: An Overview,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 14, no. 5, pp. 910–932, 2020.
  * [2] H. Farid, “Image forensics,” _Annual Review of Vision Science_ , vol. 5, pp. 549–573, 2019.
  * [3] G. A. Kaissis, M. R. Makowski, D. Rückert, and R. F. Braren, “Secure, privacy-preserving and federated machine learning in medical imaging,” _Nature Machine Intelligence_ , vol. 2, no. 6, pp. 305–311, 2020.
  * [4] B. Han, X. Han, H. Zhang, J. Li, and X. Cao, “Fighting Fake News: Two Stream Network for Deepfake Detection via Learnable SRM,” _IEEE Transactions on Biometrics, Behavior, and Identity Science_ , vol. 3, no. 3, pp. 320–331, 2021\.
  * [5] C. Dong, X. Chen, R. Hu, J. Cao, and X. Li, “MVSS-Net: Multi-View Multi-Scale Supervised Networks for Image Manipulation Detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–14, 2022.
  * [6] S. Li, S. Xu, W. Ma, and Q. Zong, “Image Manipulation Localization Using Attentional Cross-Domain CNN Features,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–15, 2021.
  * [7] Y. Wu, W. AbdAlmageed, and P. Natarajan, “ManTra-Net: Manipulation tracing network for detection and localization of image forgeries with anomalous features,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9543–9552.
  * [8] X. Hu, Z. Zhang, Z. Jiang, S. Chaudhuri, Z. Yang, and R. Nevatia, “SPAN: Spatial pyramid attention network for image manipulation localization,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 312–328.
  * [9] Y. Rao and J. Ni, “Self-supervised domain adaptation for forgery localization of JPEG compressed images,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2021, pp. 15 034–15 043.
  * [10] P. Zhou, X. Han, V. I. Morariu, and L. S. Davis, “Learning Rich Features for Image Manipulation Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1053–1061.
  * [11] Y. Ji, H. Zhang, Z. Zhang, and M. Liu, “CNN-based encoder-decoder networks for salient object detection: A comprehensive review and recent advances,” _Information Sciences_ , vol. 546, pp. 835–857, 2021.
  * [12] L. Zhuo, S. Tan, B. Li, and J. Huang, “Self-Adversarial Training Incorporating Forgery Attention for Image Forgery Localization,” _IEEE Transactions on Information Forensics and Security_ , vol. 17, pp. 819–834, 2022\.
  * [13] P. Zhuang, H. Li, S. Tan, B. Li, and J. Huang, “Image tampering localization using a dense fully convolutional network,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 2986–2999, 2021.
  * [14] Y. Rao, J. Ni, and H. Xie, “Multi-semantic CRF-based attention model for image forgery detection and localization,” _Signal Processing_ , vol. 183, p. 108051, 2021.
  * [15] X. Bi, Z. Zhang, and B. Xiao, “Reality Transform Adversarial Generators for Image Splicing Forgery Detection and Localization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2021, pp. 14 294–14 303.
  * [16] X. Wang, Y. Wang, J. Lei, B. Li, Q. Wang, and J. Xue, “Coarse-to-fine-grained method for image splicing region detection,” _Pattern Recognition_ , vol. 122, p. 108347, 2022.
  * [17] Y. Niu, B. Tondi, Y. Zhao, R. Ni, and M. Barni, “Image Splicing Detection, Localization and Attribution via JPEG Primary Quantization Matrix Estimation and Clustering,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 5397–5412, 2021.
  * [18] Y. Liu, C. Xia, X. Zhu, and S. Xu, “Two-Stage Copy-Move Forgery Detection With Self Deep Matching and Proposal SuperGlue,” _IEEE Transactions on Image Processing_ , vol. 31, pp. 541–555, 2022.
  * [19] Y. Zhu, C. Chen, G. Yan, Y. Guo, and Y. Dong, “AR-Net: Adaptive Attention and Residual Refinement Network for Copy-Move Forgery Detection,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 10, pp. 6714–6723, 2020\.
  * [20] A. Islam, C. Long, A. Basharat, and A. Hoogs, “DOA-GAN: Dual-order attentive generative adversarial network for image copy-move forgery detection and localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4676–4685.
  * [21] J. Wang, Z. Wu, J. Chen, X. Han, A. Shrivastava, S.-N. Lim, and Y.-G. Jiang, “Objectformer for image manipulation detection and localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2364–2373.
  * [22] M.-J. Kwon, S.-H. Nam, I.-J. Yu, H.-K. Lee, and C. Kim, “Learning JPEG Compression Artifacts for Image Manipulation Detection and Localization,” _International Journal of Computer Vision_ , vol. 130, p. 1875–1895, 2022\.
  * [23] J. H. Bappy, C. Simons, L. Nataraj, B. S. Manjunath, and A. K. Roy-Chowdhury, “Hybrid LSTM and Encoder–Decoder Architecture for Detection of Image Forgeries,” _IEEE Transactions on Image Processing_ , vol. 28, no. 7, pp. 3286–3300, 2019.
  * [24] J. Hu, L. Shen, and G. Sun, “Squeeze-and-Excitation Networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7132–7141.
  * [25] A. G. Roy, N. Navab, and C. Wachinger, “Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks,” in _Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention_ , 2018, pp. 421–429.
  * [26] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “CBAM: Convolutional block attention module,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 3–19.
  * [27] J. Park, S. Woo, J.-Y. Lee, and I. S. Kweon, “BAM: Bottleneck attention module,” _arXiv preprint arXiv:1807.06514_ , 2018.
  * [28] Z. Gao, J. Xie, Q. Wang, and P. Li, “Global second-order pooling convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3024–3033.
  * [29] Y. Li, Q. Miao, W. Ouyang, Z. Ma, H. Fang, C. Dong, and Y. Quan, “LAP-Net: Level-aware progressive network for image dehazing,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3276–3285.
  * [30] Y. Mao, Q. Jiang, R. Cong, W. Gao, F. Shao, and S. Kwong, “Cross-Modality Fusion and Progressive Integration Network for Saliency Prediction on Stereoscopic 3D Images,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 2435–2448, 2021.
  * [31] H. Chen and Y. Li, “Progressively complementarity-aware fusion network for RGB-D salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3051–3060.
  * [32] X. Liu, Y. Liu, J. Chen, and X. Liu, “PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 11, pp. 7505–7517, 2022.
  * [33] Y. Sun, R. Ni, and Y. Zhao, “ET: Edge-Enhanced Transformer for Image Splicing Detection,” _IEEE Signal Processing Letters_ , vol. 29, pp. 1232–1236, 2022\.
  * [34] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-Supervised Nets,” in _Proceedings of International Conference on Artificial Intelligence and Statistics_ , vol. 38, 2015, pp. 562–570.
  * [35] X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand, “BASNet: Boundary-Aware Salient Object Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7479–7489.
  * [36] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” _Annals of Operations Research_ , vol. 134, no. 1, pp. 19–67, 2005.
  * [37] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-scale Image Recognition,” _arXiv preprint arXiv:1409.1556_ , 2014\.
  * [38] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet Large Scale Visual Recognition Challenge,” _International Journal of Computer Vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [39] Z. Gao, C. Sun, Z. Cheng, W. Guan, A. Liu, and M. Wang, “TBNet: A Two-Stream Boundary-Aware Network for Generic Image Manipulation Localization,” _IEEE Transactions on Knowledge and Data Engineering_ , pp. 1–16, 2022.
  * [40] N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete Cosine Transform,” _IEEE Transactions on Computers_ , vol. 100, no. 1, pp. 90–93, 1974.
  * [41] B. Bayar and M. C. Stamm, “Constrained Convolutional Neural Networks: A New Approach Towards General Purpose Image Manipulation Detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 13, no. 11, pp. 2691–2706, 2018.
  * [42] J. Fridrich and J. Kodovsky, “Rich Models for Steganalysis of Digital Images,” _IEEE Transactions on Information Forensics and Security_ , vol. 7, no. 3, pp. 868–882, 2012.
  * [43] H. Li, G. Chen, G. Li, and Y. Yu, “Motion Guided Attention for Video Salient Object Detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7274–7283.
  * [44] J. Cao, C. Ma, T. Yao, S. Chen, S. Ding, and X. Yang, “End-to-End Reconstruction-Classification Learning for Face Forgery Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4113–4122.
  * [45] Y. Zhang, G. Zhu, L. Wu, S. Kwong, H. Zhang, and Y. Zhou, “Multi-task SE-Network for Image Splicing Localization,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 7, pp. 4828–4840, 2022\.
  * [46] H. Wu and J. Zhou, “IID-Net: Image Inpainting Detection Network via Neural Architecture Search and Attention,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 3, pp. 1172–1185, 2022.
  * [47] J. Shore and R. Johnson, “Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-entropy,” _IEEE Transactions on Information Theory_ , vol. 26, no. 1, pp. 26–37, 1980.
  * [48] X. Qin, D.-P. Fan, C. Huang, C. Diagne, Z. Zhang, A. C. Sant’Anna, A. Suarez, M. Jagersand, and L. Shao, “Boundary-aware segmentation network for mobile and web applications,” _arXiv preprint arXiv:2101.04704_ , 2021.
  * [49] J. Yu, Y. Jiang, Z. Wang, Z. Cao, and T. Huang, “Unitbox: An Advanced Object Detection Network,” in _Proceedings of the ACM International Conference on Multimedia_ , 2016, pp. 516–520.
  * [50] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: from Error Visibility to Structural Similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [51] J. Dong, W. Wang, and T. Tan, “CASIA Image Tampering Detection Evaluation Database,” in _Proceedings of IEEE China Summit and International Conference on Signal and Information Processing_ , 2013, pp. 422–426.
  * [52] T.-T. Ng, J. Hsu, and S.-F. Chang, “Columbia Image Splicing Detection Evaluation Dataset,” _DVMM lab. Columbia Univ CalPhotos Digit Libr_ , 2009\.
  * [53] T. J. De Carvalho, C. Riess, E. Angelopoulou, H. Pedrini, and A. de Rezende Rocha, “Exposing Digital Image Forgeries by Illumination Color Classification,” _IEEE Transactions on Information Forensics and Security_ , vol. 8, no. 7, pp. 1182–1194, 2013.
  * [54] Y. Wu, W. Abd-Almageed, and P. Natarajan, “BusterNet: Detecting copy-move image forgery with source/target localization,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 168–184.
  * [55] D. Tralic, I. Zupancic, S. Grgic, and M. Grgic, “CoMoFoD-New Database For Copy-Move Forgery Detection,” in _Proceedings of ELMAR_ , 2013, pp. 49–54.
  * [56] B. Wen, Y. Zhu, R. Subramanian, T.-T. Ng, X. Shen, and S. Winkler, “COVERAGE-A Novel Database for Copy-Move Forgery Detection,” in _Proceedings of IEEE International Conference on Image Processing_ , 2016, pp. 161–165.
  * [57] R. Salloum, Y. Ren, and C.-C. J. Kuo, “Image Splicing Localization using a Multi-task Fully Convolutional Network (MFCN),” _Journal of Visual Communication and Image Representation_ , vol. 51, pp. 201–209, 2018.
  * [58] J. H. Bappy, A. K. Roy-Chowdhury, J. Bunk, L. Nataraj, and B. Manjunath, “Exploiting Spatial Structure for Localizing Manipulated Image Regions,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4970–4979.
  * [59] X. Cun and C.-M. Pun, “Image Splicing Localization via Semi-Global Network and Fully Connected Conditional Random Fields,” in _Proceedings of the European Conference on Computer Vision Workshops_ , 2018, pp. 1–15.
  * [60] X. Bi, Y. Wei, B. Xiao, and W. Li, “RRU-Net: The Ringed Residual U-Net for Image Splicing Forgery Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 1–10.
  * [61] B. Xiao, Y. Wei, X. Bi, W. Li, and J. Ma, “Image Splicing Forgery Detection Combining Coarse to Refined Convolutional Neural Network and Adaptive Clustering,” _Information Sciences_ , vol. 511, pp. 172–191, 2020.
  * [62] Y. Wei, Z. Wang, B. Xiao, X. Liu, Z. Yan, and J. Ma, “Controlling Neural Learning Network with Multiple Scales for Image Splicing Forgery Detection,” _ACM Transactions on Multimedia Computing, Communications, and Applications_ , vol. 16, no. 4, pp. 1–22, 2020.
  * [63] B. Chen, W. Tan, G. Coatrieux, Y. Zheng, and Y.-Q. Shi, “A Serial Image Copy-Move Forgery Localization Scheme With Source/Target Distinguishment,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 3506–3517, 2021.


# The use of new technologies to support Public Administration.
Sentiment analysis and the case of the app ’IO’

Vincenzo Miracula
Dipartimento di Fisica e Astronomia “E. Majorana”
Università di Catania
Catania
<EMAIL_ADDRESS>
Antonio Picone
Dipartimento di Fisica e Astronomia “E. Majorana”
Università di Catania
Catania
<EMAIL_ADDRESS>

###### Abstract

App IO is an app developed for the Italian PA. It is definitely useful for
citizens to interact with the PA and to get services that were not digitized
yet. Nevertheless, it was not perceived in a good way by the citizens and it
has been criticized. As we wanted to find the root that caused all these bad
reviews we scraped feedback from mobile app stores using custom-coded
automated tools and - after that - we trained two machine learning models to
perform both sentiment analysis and emotion detection to understand what
caused the bad reviews.

_K_ eywords Public Administration, Artificial Intelligence, Digital
Transformation, Decision Making

##  1 Introduction

Since 2005, there has been an increasing development of digitization within
the public administration that sees the introduction of the use of technology
as a privileged tool in the management of administrative activities. The main
objective is to promote digitization in administrations in order to achieve
greater efficiency in their activities in internal relations, between
different administrations, and between the latter and private individuals. The
entry of artificial intelligence into public action, however, needs to be
accompanied by an adequate regulatory framework to guarantee the rights of
those administered.

The notion of digital transformation has gained significant attention in the
literature[1]. Although approaches to the definition of digital transformation
vary[2], most authors suggest that digital transformation involves the use of
ICT technology to create fundamentally new capabilities in business, public
administration[3] and people’s lives[4]. The theory emphasizes the importance
of digitization to optimize the public value of government services for
citizens[5] as well as to increase the efficiency of government functions by
implementing lean government models[6]. Digital technologies stimulate citizen
participation[7] and support customer involvement in the co-production and co-
creation of public value[8]. The digital transformation thus has a significant
impact not only on the accessibility and quality of public services but also
on the way other functions of public administration are carried out[9] (like
policy development, regulation, and enforcement, etc.).

A strong emphasis in the current phase is placed on government solutions as a
platform[10] and, consequently, on data-driven governance. In the academic
literature, there are also other approaches to the staging of digital
transformation in public administration. Janowski[11], for instance, suggests
that digital transformation is not a one-off event or project with a clear
start and end date, but rather an ongoing process involving changes both in
internal processes and procedures and in the way the public administration
communicates with its main beneficiaries.

It is assumed that the results of this process will be significant, but so far
no uniform approaches have been implemented to measure these results. Although
digital transformation is an evolving process, the measures used to evaluate
this process should also evolve according to the stage of the digital
transformation. For instance, in the early stages of e-government, indicators
such as the ratio of e-services provided online are considered relevant.

In the more mature stages of digital transformation, the effects are
different: smart government leads to the replacement of official portals by
automated interactions and, thus, results in the reduction of the types of
public services[9]. This contribution intends to present new tools for
measuring the results of digital transformation through the use of sentiment
analysis and social network analysis.

##  2 The use of the IO app in public administration

As of 18 April 2020, the IO app is downloadable from the online stores for
Android and Apple. The app, conceived and developed by the Digital
Transformation Team, was still in Beta version in April 2020, with a limited
number of services available, but making it downloadable was an important
first step to start testing it on a large scale. The IO app, in fact, is the
platform designed to allow all citizens to have a new and single telematic
access point to the services, information, and communications of the public
administration, thus enabling them to use national and local public services
from their smartphones in a simple, modern and secure way.

Two years after its release on online stores, there have been many
developments, mainly related to the fact that the app was chosen as a tool to
access some of the services activated in the months of the Covid-19 pandemic,
such as the holiday bonus, the ’Cashback’ program and the green-pass. The
regulatory basis of the IO project is Art. 64 bis of the new Digital
Administration Code[12] (CAD), which provides a single point of telematic
access to public administration services and highlights a fundamental change
in the relationship between citizens and PA, centered on three key aspects:
simplicity, speed, and transparency. The ultimate goal is to bring the citizen
closer to the administration, making simple mechanisms that often still prove
to be complex and cumbersome. The IO app is, therefore, the tool designed to
concretely enable Digital Citizenship, providing citizens with a direct
connection (through their smartphone) with PA services and communications, a
single access point for the provision and use of public services.

As can be understood, for now, it is always the citizen who has to express his
or her willingness to interact with the Public Administration through the IO
app. By using the ’IO’ app, the citizen agrees to interface with the Public
administration he or she needs to get in touch with according to the
provisions of the 2019 Three-Year Plan for Information Technology in Public
Administration.

In the document ’Strategy for Technological Innovation and the Digitisation of
the Country’[13] (also called, Plan 2025) presented on 17 December 2019 by
Minister Pisano, the IO app is included among the top 20 Actions to Transform
the Country. In fact, page 15 of the document states that "Italy, at least
digitally, should be one big municipality that treats all its citizens
equally", and again, "IO, is the public services app that transforms the
relationship between citizens and Public Administration, putting people at the
center and erasing complexity: a single interface to access all public
services directly from your smartphone after identifying yourself with your
digital identity."

##  3 Methodology

In order to extract data for the purposes of our analysis, we decided to
extract user reviews from the two main stores for mobile devices[14]: AppStore
for Apple iOS users and Play Store for Android users. To date, both account
for 95% of total downloads, so they constitute a representative sample of the
thinking of the users of such applications in the Italian context.

In order to extract the reviews, code was written in Python, so as to automate
the extraction of the reviews, not only of the text but also of the author of
the review and the stars that were assigned (from 1 to 5). This process is
called ’scraping’, an English term that literally means ’scraping’. Since
there is no official API, which would allow programmatic access, the only
alternative for data extraction is scraping. This technique consists of
accessing the various tags of the HTML source code in order to extract the
desired fields. This approach certainly requires higher skills than the use of
an API and a further cleaning of the data, sometimes soiled by the presence of
HTML tags that must be removed, in order to obtain text in its classic form.

The dataset, consisting of the cleaned textual reviews and the metadata
mentioned, was saved in serialized JSON (JavaScript Object Notation) format,
for easy reading by both a human being and a computer, as further analysis of
sentiment and emotions followed, obtained through the use of machine learning
techniques applied to Natural Language Processing. In particular, BERT
(Bidirectional Encoder Representations from Transformers) was used, a type of
neural network, based on Transformers, devised by Google[15], thanks to whose
attention mechanism, each word within the same sentence is related both to the
word that follows it, and to the previous one, allowing the entire context of
a sentence to be maintained.

We therefore, on the basis of an existing model for Italian, carried out what
is called fine-tuning, i.e. starting from a general model, not specific to any
task, the model was trained with our data to perform and perform better in the
two tasks specific to our objective. In this way, we obtained a sentiment
analysis model capable of discerning sentences into two categories: positive
and negative; furthermore, as already mentioned, a model for emotion detection
was also realized. There has long been debate as to which emotions are worth
training our model on it is necessary to have a psychological background on
the matter and the literature on the subject has proved to be of fundamental
importance. Paul Ekman[16] is an American psychologist who came up with the
universal theory of emotions, according to which there are seven primary
emotions: fear, anger, joy, sadness, contempt, disgust, and surprise; but
these have been reformulated and amalgamated to become six in its latest
official version, namely: fear, anger, joy, sadness, disgust, and surprise.
However, according to a recent study[17], it is proposed that human beings
have four basic emotions: fear, anger, joy, and sadness. Over the course of
time, other authors have also proposed the above emotions as the four basic
emotions[18],[19]. In light of these studies, our model follows in the
footsteps and ideas of the most recent studies on the subject, thus managing
to categorize input sentences into the four categories of emotions mentioned
above.

##  4 Network analysis

Social media platforms have become key channels for communication and
information dissemination, making it increasingly important to understand how
information spreads within these networks. In general, in social network
analysis, nodes are people and ties are all the social connections between
them - for example, friendship, marital/family ties, or financial ties.

Social network analysis (SNA) is a field of study involving the use of
statistical and mathematical techniques to analyze and understand
relationships and patterns within a network of individuals or organizations.
It is often used to identify key actors and understand the dynamics of social,
professional, and communication networks. The objective of social network
analysis is to understand a community by mapping the relationships that
connect it as a network and then trying to identify key individuals and groups
within the network and/or associations between individuals.

Twitter is an excellent platform to observe these behaviors. For this reason,
we decided to collect tweets to try to understand what are the dynamics that
lead to the formation of communities on Twitter, mapping the relationships
that connect people as a network and then trying to bring out the key
individuals, and groups within the network and associations between
individuals.

In an SNA, a network is typically represented as a graph, with nodes
representing individual actors and links representing relationships between
them (e.g. friendship, collaboration). Several metrics can be used to analyze
these networks, including measures of centrality (e.g. degree) measures of
community structure (e.g. modularity), and measures of network evolution (e.g.
preferential attachment).

SNA has a wide range of applications, including studying the spread of
information and ideas, identifying influencers, understanding the structure
and dynamics of social networks, and predicting the formation of new
relationships. It is used in fields such as sociology, psychology,
anthropology, communication, and information technology and has also been
applied to the analysis of networks in business, politics, and public health.

##  5 Results

Social networks represent an emerging challenging sector where the natural
language expressions of people can be easily reported through short but
meaningful text messages. Key information that can be grasped from social
environments relates to the polarity of text messages.

The first result of our research is related to the citizen’s perception of the
IO app. We then extracted 62986 reviews from the various stores and through
the use of Natural Language Processing[20] techniques such as sentiment
analysis and emotion detection, we have that 75.91% of the reviews have
negative sentiment and 60.5% negative emotions (34.3% sadness, 26.2% anger).

The second result relates to SNA. From our analysis, the SNA presents 5
different and distinct communities calculated according to Louvain’s
algorithm[21]. The community detection thus constructed shows moderately
connected communities with a modularity value of 0.6. We also note that the
graph presents itself according to a Barabasi-Albert[22] with 62986 nodes with
an average degree (i.e. the average number of links) of 1.37 and a density of
0.573(1).

(H)  Statistics Value Number of nodes 62986 Mean degree 1.37 Density 0.573
Modularity 0.6

Table 1: Network results

These results, as shown in Fig. 1 lead to the hypothesis that individuals
struggle to change their opinion and sentiment toward the use of the IO app
(and by extension towards the digitization path undertaken by the PA).

Figure 1: network analysis IO App.

##  6 Conclusion

This paper brings attention to how computational social science and network
science can both explain the complex dynamics of controversial and challenging
topics. The digital ecosystem not only evolves social network communication
but also provides the social researcher with useful data to explain social-
complex dynamics.

Using natural language processing techniques, such as sentiment analysis and
emotion detection, we proposed a new way to measure the impact of digital
transformation and how the resulting analysis can be applied to provide
legislators with meta-evaluation tools, in this case starting with how the app
is perceived by citizens.

We noticed that many of the comments found within the dataset we created are
not related to the app itself, a large number of them are negative towards the
government and the PA itself. The comments are therefore just a proxy for the
citizen to criticize policy choices.

## References

  * [1] Vesna Bosilj Vukšić, Lucija Ivančić, and Dalia Suša Vugec.  A preliminary literature review of digital transformation case studies.  International Journal of Computer and Information Engineering, 12(9):737–742, 2018.
  * [2] João Reis, Marlene Amorim, Nuno Melão, and Patrícia Matos.  Digital transformation: a literature review and guidelines for future research.  In World conference on information systems and technologies, pages 411–421. Springer, 2018.
  * [3] Colin Lankshear and Michele Knobel.  Digital literacies: Concepts, policies and practices, volume 30.  Peter Lang, 2008.
  * [4] George Westerman, Claire Calméjane, Didier Bonnet, Patrick Ferraris, Andrew McAfee, et al.  Digital transformation: A roadmap for billion-dollar organizations.  MIT Center for digital business and capgemini consulting, 1:1–68, 2011.
  * [5] Frank Bannister and Regina Connolly.  Ict, public values and transformative government: A framework and programme for research.  Government Information Quarterly, 31(1):119–128, 2014.
  * [6] Marijn Janssen and Elsa Estevez.  Lean government and platform-based governance—doing more with less.  Government Information Quarterly, 30:S1–S8, 2013.
  * [7] Luis Felipe Luna-Reyes.  Opportunities and challenges for digital governance in a world of digital participation.  Information polity, 22(2-3):197–205, 2017.
  * [8] Antonio Cordella and Andrea Paletti.  Icts and value creation in public sector: Manufacturing logic vs service logic.  Information Polity, 23(2):125–141, 2018.
  * [9] Elena Dobrolyubova.  Measuring outcomes of digital transformation in public administration: Literature review and possible steps forward.  NISPAcee Journal of Public Administration and Policy, 14(1):61–86, 2021.
  * [10] Tim O’Reilly.  Government as a platform.  Innovations: Technology, Governance, Globalization, 6(1):13–40, 2011.
  * [11] Tomasz Janowski.  Digital government evolution: From transformation to contextualization, 2015.
  * [12] Consiglio dei Ministri.  Codice dell’amministrazione digitale, 2022.
  * [13] Ministro per l’Innovazione tecnologica e la digitalizzazione.  Piano nazionale innovazione 2025, 2022.
  * [14] C Scott Hemphill.  Intellectual property and competition law.  Forthcoming, Oxford Handbook of Intellectual Property Law (Rochelle C. Dreyfuss & Justine Pila eds. 2017), 2017.
  * [15] Ulrike von Luxburg, I Guyon, S Bengio, H Wallach, R Fergus, SVN Vishwanathan, and R Garnett.  Advances in neural information processing systems 30.  In 31st annual conference on neural information processing systems (NIPS 2017), Long Beach, California, USA, pages 4–9, 2017.
  * [16] Paul Ekman.  An argument for basic emotions.  Cognition & emotion, 6(3-4):169–200, 1992.
  * [17] Rachael E Jack, Oliver GB Garrod, and Philippe G Schyns.  Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time.  Current biology, 24(2):187–192, 2014.
  * [18] Simeng Gu, Fushun Wang, Tifei Yuan, Benyu Guo, and Jason H Huang.  Differentiation of primary emotions through neuromodulators: review of literature.  International Journal of Neurology Research, 1(2):43–50, 2015.
  * [19] Fushun Wang and Alfredo Pereira.  Neuromodulation, emotional feelings and affective disorders.  Mens sana monographs, 14(1):5, 2016.
  * [20] KR1442 Chowdhary.  Natural language processing.  Fundamentals of artificial intelligence, pages 603–649, 2020.
  * [21] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre.  Fast unfolding of communities in large networks.  Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
  * [22] Réka Albert and Albert-László Barabási.  Statistical mechanics of complex networks.  Reviews of modern physics, 74(1):47, 2002.