# Manipulation Mask Generator: High-Quality Image Manipulation Mask Generation
Method Based on Modified Total Variation Noise Reduction

1st Xinyu Yang  School of Computer Science
Sichuan University
Chengdu, China
<EMAIL_ADDRESS>2nd Jizhe Zhou  School of Computer Scienc
Sichuan University
Chengdu, China
<EMAIL_ADDRESS>
###### Abstract

In artificial intelligence, any model that wants to achieve a good result is
inseparable from a large number of high-quality data. It is especially true in
the field of tamper detection. This paper proposes a modified total variation
noise reduction method to acquire high-quality tampered images. We
automatically crawl original and tampered images from the Baidu PS Bar. Baidu
PS Bar is a website where net friends post countless tampered images.
Subtracting the original image with the tampered image can highlight the
tampered area. However, there is also substantial noise on the final print, so
these images can’t be directly used in the deep learning model. Our modified
total variation noise reduction method is aimed at solving this problem.
Because a lot of text is slender, it is easy to lose text information after
the opening and closing operation. We use MSER (Maximally Stable Extremal
Regions) and NMS (Non-maximum Suppression) technology to extract text
information. And then use the modified total variation noise reduction
technology to process the subtracted image. Finally, we can obtain an image
with little noise by adding the image and text information. And the idea also
largely retains the text information. Datasets generated in this way can be
used in deep learning models, and they will help the model achieve better
results.

###### Index Terms:

total variation, automated data crawling, maximally stable extremal regions

##  I INTRODUCTION

With the advances in image editing techniques and user-friendly editing
software, low-cost tampered or manipulated image generation processes have
become widely available[4]. Non-professional users can easily edit and tamper
images without leaving obvious visual traces[5]. Some tampered images are
amusing and harmless, But some can damage the reputation of others and even
spread rumors to cause panic. Meanwhile, traditional detection methods play a
limited role in processing these images. Therefore, deep learning is more
needed in this realm. On the other hand, deep learning models have
demonstrated their power in many applications. For example, convolution neural
networks (CNNs) achieve promising performance in many computer vision and
natural language processing applications[6].
Deep learning provides a novel approach to identifying features for tempered
regions, which inherently represent characteristics of the tempered regions
appearing in the dataset[14]. Compared to traditional image detection methods,
deep learning models can automatically learn the features and patterns of
images from a large amount of data and can more accurately recognize and
classify these features and designs, thus achieving higher accuracy. And they
can learn more complex features and patterns, thus better coping with various
deformations and distortions in image tampered, improving the robustness and
stability of the algorithm. Also, Deep learning models can increase the number
of layers and parameters to improve the complexity and performance of the
algorithm, thus better adapting to different image-tampered scenarios and
application requirements. Besides, Deep learning models automatically warn
parameters according to different tampered scenarios and datasets, thus better
adapting to various image-tampered detection tasks. Meanwhile, faster
processing speed is also one of its advantages: they can use high-performance
hardware such as GPUs to accelerate computing, thus achieving faster image-
tampered detection speed to meet the requirements of real-time and high
efficiency.
In image-tampered detection, deep learning requires a large number of high-
quality datasets for training and validation. Large-scale, high-quality
datasets play an essential role in the deep learning era, which act as the
catalyst stimulating and accelerating technique development[18]. However, Some
databases for CMFD(copy-move forgery detection) already exist, but they are
not suited for evaluating post-processing methods[15]. Firstly, current
datasets suffer from homogeneity and insufficient sample size, making it
difficult for deep learning algorithms to predict new tampered techniques in
practical applications. Secondly, existing datasets are often generated
through simulated tampered behavior, which may only partially reflect the
tampered situation in real-world scenarios. Therefore, collecting and
annotating real-world datasets is vital but requires significant time and
cost.

TABLE I: The existing tamper detection data sets Existing DataSets | Shortcomings
---|---
Defacto | random synthesize
CASIA v1.0 | monotonous tamper types
NIST 2016 | label leakage
COVERAGE | small sample size
CASIA v2.0 | unspecified enhancement

We list the five main datasets and their shortcomings in Table 1. DFACTO is a
novel dataset for image and face manipulation detection and localization[17].
CASIA v1.0 is one of the more commonly used datasets for image tampered
detection, containing a small number of manually tampered images and
synthetically tampered images generated by copying and pasting. The database
is made publicly available for researchers to compare and evaluate their
proposed tampering detection techniques[12]. However, the sample size is
relatively small, and covering all possible tampered scenarios is difficult.
NIST 2016 was used in the image tampered detection competition held by the
National Institute of Standards and Technology (NIST) in 2016. It contains a
large number of tampered images and real-world images, but it is private, and
only competition participants can use it, making it difficult for researchers
to access. Although COVERAGE includes both manually tampered images and real-
world images, the dataset is still being expanded, and the sample size is
relatively small, making it challenging to meet the needs of deep learning
algorithms. CASIA v2.0 is a large-scale face dataset released by the Chinese
Academy of Sciences. The datasets still had some problems, such as imbalanced
and limited subject distribution, image quality and metadata, and Potential
for overfitting. These issues may limit the generalization ability of machine
learning models trained solely on CASIA v2.0. We will show our results in
Fig.1.

Figure 1: We list four groups of images, from left to right, the original
image, the tampered image, the noise image after subtracting the two images,
and finally, our processing result image. We can see that our method
accurately finds the tampered area. The meaning of Chinese characters: pay 50¥
to remove the watermark.

Considering the increasing demand for public databases for image forensic
[12], we crawl images from websites with tampered images on the network, such
as the Baidu PS Bar. It is a perfect source of original and tampered images.
Most users will ask others to help them modify the picture they offer. In that
case, Under their posts, there are often a large number of tampered images. We
need to save the original and tampered images from different posts. That makes
us collect lots of data in a short time. The usual way to find the difference
between the tampered image and the original image is to subtract the tampered
graph from the original one. However, the user, in the process of modifying
the image, tried to make a universe change to the image. That will make the
result of subtraction have a lot of noise, even full-screen noise. Using total
variation noise reduction can effectively solve this problem. This method
combines character recognition technology and some switching operations, which
can significantly retain text information and reduce noise.
In general, the contribution of our paper contains:

  * •

We propose a modified total variation noise reduction method. Total variation
noise reduction can remove various types of noise, which will help us obtain a
high-quality mask.

  * •

We design a way to automatically obtain a large number of datasets, which will
help us quickly get a large number of available data.

Figure 2:  Modified total variation noise reduction architecture to obtain
high-quality mask. Many original data is crawled from websites containing
tampered images by crawlers. The original and tampered images are converted
into gray images and then subtracted. Contour detection is used to extract the
text part, and the gray image will be iterated one hundred times by total
variation noise reduction. Then the image will be binarized and processed by
switching operations. Words part and binary image add up to a high-quality
mask.

##  II RELATED WORKS

###  II-A Automated Data Crawling

The dataset in the image tampered detection field is usually small because
manual annotation is required. A growing wealth of information and
increasingly sophisticated interfaces necessitate automated processing[7] in
recent years. The conventional web scraping methods for data collection
include Web Crawlers, API Calls, RSS Subscriptions, and Database access. API,
or Application Programming Interface, is a set of software tools that allows
different applications to interact with each other. Through an API, we can
obtain the desired data from other applications and return the data in a
machine-readable format. RSS is a standard protocol for obtaining updates from
blogs, news websites, and other sources. You can get updated content in real
time by subscribing to RSS feeds. What we have applied is web crawlers. The
web crawler is a program or software that traverses the Web and downloads web
documents in a methodical, automated manner[8]. Crawlers usually access the
target website’s pages according to predetermined rules, extract the data and
save it locally or in a database. On the network, there are plenty of tampered
images. We need to collect tampered images and their original images to get
the data that could be used in the deep learning model. PS websites can meet
this need. The PS bar in Baidu Post is our first choice.

###  II-B Total Variation

The total variation has been introduced in Computer Vision by Rudin, Osher,
and Fatemi[3] as a regularizing criterion for solving inverse problems[9]. It
has many applications in image processing, computer vision, and image
analysis. The total variation can be understood as the L1 norm of the gradient
in the image, that is, the sum of the absolute values of the adjacent
differences of the pixel values. The smaller the total variation, the higher
the smoothness of the image. That is, the smaller the pixel value change in
the image, and vice versa. It means that there are more details and texture
information in the image. In image processing, total variation is often used
as a regularization term to constrain the solution of optimization problems,
such as image noise reduction, image segmentation, image enhancement, etc.
Total variation regularization can be achieved by adding a total variation
term to the objective optimization function. Introducing the total variation
regularization term can effectively suppress noise and protect image edge
details from obtaining better image processing results. In addition to two-
dimensional images, total variation can be extended to three-dimensional
images, video, and other signal processing problems with wide application
value. Arguably, the success of TV-based regularization lies in a good balance
between the ability to model piece-wise smooth images and the difficulty of
the resulting optimization problems[10].

##  III PROPOSED METHODS

###  III-A Maximally Stable Extremal Regions

Our general working pipeline is depicted in Fig 2. For the commencement, we
crawl tons of raw data from the website with tampered images. Only tampered
images of the same size as the original can be processed. After the two images
are subtracted and converted into a gray image, we will get an image with much
noise. Usually, words in the image are obvious but thin. They are critical
modified parts. However, processing them directly with total variation noise
reduction technology will lead to the loss of this information. So we must
pre-process the image with text recognition to retain this information.

As the intensity contrast of text to its background is typically significant,
and a uniform intensity or color within every letter can be assumed, MSER is a
natural choice for text detection[11]. The SIFT and SURF algorithms proposed
by Lowe and Bay efficiently achieve feature detection with scale and rotation
invariance, but these features are not affine-invariant. For various image
regions with different shapes, affine-invariance is achieved by region
rotation and size normalization. MSER is one of the most influential
algorithms in region detection. The concept can be explained as follows.
Imagine all possible three holdings of a gray-level image i. We will refer to
the pixels below a threshold as ’black’ and to those above or equal as
’white.’ If we were shown a movie of threshold images, with the frame
corresponding to a threshold, we would see first a white image. Subsequently,
black spots corresponding to local intensity minimal will appear and grow. At
some point, regions corresponding to two local minimal will merge. Finally,
the last image will be black. The set of all connected components of all
frames of the movie is the set of all maximal regions; minimal regions could
be obtained by inverting the intensity of I and running the same process[2].
Its mathematical principles are as follows:

$v(i)=\frac{\left|Q_{i+\Delta}-Q_{i-\Delta}\right|}{\left|Q_{i}\right|}$ (1)

###  III-B Non-maximum Suppression

We can put much-suspected text information in part of the box through MSER
technology. However, dozens of boxes will be around the same part of the
situation. These boxes overlap, which seriously interferes with our selection
of the best effective region. So we use the Non-maximum Suppression method to
choose the best one. Pedro F. Felzenszwalb[1] described an object detection
system that represents highly variable objects using mixtures of multiscale
deformable part models in 2009. These models are trained using a
discriminative procedure that only requires bounding boxes for the objects in
a set of images[1]. Their proposed technology is a component-based object
detection system that uses a hybrid representation of multiscale deformable
component models. In this detection system, NMS removes overlaps between
multiple components. We take NMS to remove overlaps, too.

In target detection, NMS is a post-processing method for removing overlapped
bounding boxes. After applying the bounding box prediction method described
above, we have a set of detection D for a particular object category in an
image. A bounding box and a score define each detection. We sort the detection
in D by score and greedily select the highest-scoring ones while skipping
detection with bounding boxes that are at least 50% covered by a bounding box
of a previously selected detection[1]. Through this theory, we can remove
overlaps like Fig. 3

Figure 3: The results of NMS. MSER will give us several rectangular frames,
some of which are too large, some are too small, and some overlap too much, as
shown in the left diagram. Our NMS technology reduces the multiple frames of
the above into one of the most representative frames, as the right image
shows.

We can obtain many bounding boxes from MSER. In terms of evaluation metrics
for bounding box regression, Intersection over Union (IoU) is the most popular
metric[16].

$IOU=\frac{A\cap B}{A\cup B}$ (2)

We set the intersection ratio $>=0.4$ to delete the comparison block diagram,
leaving the highest score block diagram. Go to the block diagram below the
threshold, sort the remaining block diagram, select the block diagram with a
high confidence value, and repeat the intersection and comparison process.

###  III-C Modified Total Variation Noise Reduction

Rudin et al.[3] observed in 1990 that the total variation of noise-
contaminated images is significantly more significant than that of noise-free
images. Total variation (TV) methods are very effective for recovering
“blocky,” possibly discontinuous, images from noisy data[13]. The total
variation is defined as the integral of the gradient amplitude. Limiting the
total variation limits the noise. The total variation of the image is
minimized subject to constraints involving the noise statistics. The
constraints are imposed using Lagrange multipliers. The solution is obtained
using the gradient-projection method. This amounts to solving a time-dependent
partial differential equation on a manifold determined by the constraints, as
$t\rightarrow\infty$, the solution converges to a steady state which is the
denoised image. The numerical algorithm is simple and relatively fast[3]. In
the data we obtain, the signal that may be false details has a high total
variation. That is, the integral of the absolute gradient of the signal is
high. According to this principle, the total change of the signal is reduced
so that it is closely matched with the original signal. The unwanted details
are removed while retaining important details, such as edges. The mathematical
definition of the total variation is as follows:

$\inf_{u}I(u),\quad I(u):=\int_{\Omega}|\nabla u|\mathrm{d}x\mathrm{~{}d}y$

However, in digital images, there is no continuous function. The whole image
is composed of pixels, and they are discrete. We use pixel $x_{t}$ and the
next pixel $x_{t+1}$ to represent $x_{t+1}\rightarrow x_{t}$. The formula is
modified like this:

$V(y)=\sum_{i,j}\sqrt{\left|y_{i+1,j}-y_{i,j}\right|^{2}+\left|y_{i,j+1}-y_{i,j}\right|^{2}}$

In this way, we can process the masks which have massive noise. For the
processed image, we calculate each pixel’s first-order and second-order
partial derivatives in the x and y directions by looping. To make the
difference between the result and the original image not too large, we add a
fidelity term when solving this gradient minimum. The value of each point is
calculated according to the formula, and then we iterate the above operation
one hundred times. The final image still can’t be directly used because it’s
even more blurred. Using two erosion operations on this image, then binarizing
it with a threshold of 15, we can get an image that has less noise. There are
still some sporadic pixels we don’t need. We can get an image with far less
noise by processing them with 8 erosion operations and 2 dilation operations.
Besides, inputting different values to the constants in the fidelity term will
result in different processed images. Adding up the white parts of the total
variation denoising image and the same image processed by MSER, we can obtain
the high quality which could be used in the deep learning model.

##  IV EXPERIMENTS AND RESULTS

###  IV-A Maximally Stable Extremal Regions Experiments and Results

Experiments are conducted on the tampered image dataset crawled from Baidu
Post. The dataset has more than 45000 images. We’ve grouped images that
crawled down from the same post into one category. The first image is regarded
as an original image by us. The left images are almost tampered images based
on it. We throw out images that aren’t the same size as the original, for most
of these images are irrelevant emoticons. MSER can help us easily box out the
words in the mask. There are 71 bounding boxes before NMS. And there are 41
bounding boxes after NMS. We can see the result in Fig.4

Figure 4: MSER can accurately frame the part with the same gray value, most of
which are text, and a few parts that are not text are other contents that have
been tampered with. We can see 71 bounding boxes in the left graph. After NMS,
there are only 41 boxes, as the right shows. The meaning of Chinese
characters: pay 50¥ to remove the watermark.

The words we boxed out are visible. In OCR image text recognition, most of the
characters can be identified. Because of the lack of tampered images with
words, we randomly selected 50 images to detect OCR text and make the results
into a table. The text size in the picture is divided into five categories in
Fig. 5.

Figure 5: The size of the character. Different font sizes will be affected to
varying degrees in the subtraction of the two images. As shown above, we
divide the font size into five specifications and calculate the accuracy of
our method in different font sizes.

We will show the rate of text recognition in the following table II.

TABLE II: Text recognition rate Body Size | Recognition Rate
---|---
tiny | 40.3%
small | 64.7%
normalsize | 85.4%
large | 91.2%
huge | 92.4%

The tiny and small section rate is low because, in the tampered graph, small
words are greatly affected by tampering. This kind of word will be
significantly deformed after the subtraction operation, resulting in not being
recognized by OCR. For larger fonts, only a few extremely deformed characters
can not be recognized. But all the characters are converted into white parts
in the final mask. Even the fonts that cannot be identified can be marked by
our algorithm.

###  IV-B Modified Total Variation Noise Reduction Experiments and Results

After extracting the required text information, we continue to perform total
variation noise reduction on the original image. We combine the graphic part
obtained by the total variation noise reduction processing with the text part
obtained by the MSER processing. The results were corroded twice to obtain the
final results. We deploy the program on our server for 24-hour automatic image
acquisition and processing to get high-quality masks. Part of the final
results are shown in Figure 6.

Figure 6: The final result graph was obtained by combining the text part and
the graphic part. The meaning of Chinese characters: pay 50¥ to remove the
watermark.

Obviously, our operation greatly reduces the noise and retains the text
information so that the deep learning model can use the mask.

###  IV-C Advantages compared to existing methods

  * •

Large-Scale. Our raw data contains 46509 images, while the commonly used
tampering detection dataset CASIA v1.0, CASIA v2.0, Columbia, COLUMB,
FORENSICS only contains 1725, 12323, 1845, 358, and 288 images, respectively.
Such a larger scale can help better exploit the full potentials of more
advanced model architectures[19].

  * •

Continuous Regeneration. Commonly used data uses random Gaussian noise, JPRG
compression, and random flipping to make the data set one more, restricting
the increase in the data. On the contrary, net friends could supply countless
raw data for us. What we need to do is crawl their images on the internet.

  * •

Covering a Wide Range. Many datasets have their particular limitation: The
splicing forgery region of the CASIA data set is a small and delicate object.
The splicing forgery region of COLUMB is superficial, large, and meaningless.
But the dataset we build has a wide range. The types of tamper depend on how
many netizens can offer.

  * •

High-Speed Generation. Our experiments are on a 16g RAM, i7-11800H CPU, RTX
3060 GPU machine. An average size image takes around 3 minutes to process into
our mask. The large image needs more time, and the small image only needs less
than 1 minute. In this way, we can automatically get a lot of data in a very
short time. Our work efficiency will increase exponentially if multiple
devices are processed in parallel.

##  V CONCLUSIONS

In this paper, we build the Modified Total Variation Noise Reduction
architecture to automatically crawl raw data from the Internet and make it a
high-quality mask for deep learning. This technology that can obtain many
high-quality masks quickly is expected to help our deep learning model achieve
better learning results. We want to apply this technology to tampering
detection, which requires many datasets. If the deep learning model has enough
data, the harm of tampering with images in the future may be significantly
reduced.

##  VI ACKNOWLEDGEMENT

Numerical computations of this paper are supported by Hefei advanced computing
center.

## References

  * [1] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part-based models,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 32, no. 9, pp. 1627–1645, 2010.
  * [2] J. Matas, O. Chum, M. Urban, and T. Pajdla, “Robust Wide Baseline Stereo from Maximally Stable Extremal Regions.”
  * [3] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” _Physica D: Nonlinear Phenomena_ , vol. 60, no. 1-4, pp. 259–268, Nov. 1992. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/016727899290242F
  * [4] P. Zhou, X. Han, V. I. Morariu, and L. S. Davis, “Learning rich features for image manipulation detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [5] C. Beijing, J. Xingwang, G. Ye, and W. Jinwei, “A quaternion two-stream r-cnn network for pixel-level color image splicing localization,” _Chinese Journal of Electronics_ , vol. 30, no. 6, pp. 1069–1079, 2021.
  * [6] S. Zhang, H. Tong, J. Xu, and R. Maciejewski, “Graph convolutional networks: a comprehensive review,” _Computational Social Networks_ , vol. 6, no. 1, pp. 1–23, 2019.
  * [7] T. Furche, G. Gottlob, G. Grasso, C. Schallhart, and A. Sellers, “Oxpath: A language for scalable data extraction, automation, and crawling on the deep web,” _The VLDB Journal_ , vol. 22, pp. 47–72, 2013.
  * [8] M. A. Kausar, V. Dhaka, and S. K. Singh, “Web crawler: a review,” _International Journal of Computer Applications_ , vol. 63, no. 2, pp. 31–36, 2013.
  * [9] A. Chambolle, “An algorithm for total variation minimization and applications,” _Journal of Mathematical imaging and vision_ , vol. 20, pp. 89–97, 2004.
  * [10] J. P. Oliveira, J. M. Bioucas-Dias, and M. A. Figueiredo, “Adaptive total variation image deblurring: a majorization–minimization approach,” _Signal processing_ , vol. 89, no. 9, pp. 1683–1693, 2009.
  * [11] H. Chen, S. S. Tsai, G. Schroth, D. M. Chen, R. Grzeszczuk, and B. Girod, “Robust text detection in natural images with edge-enhanced maximally stable extremal regions,” in _2011 18th IEEE international conference on image processing_. IEEE, 2011, pp. 2609–2612.
  * [12] J. Dong, W. Wang, and T. Tan, “Casia image tampering detection evaluation database,” in _2013 IEEE China summit and international conference on signal and information processing_. IEEE, 2013, pp. 422–426.
  * [13] “Iterative methods for total variation denoising,” _SIAM Journal on Scientific Computing_ , vol. 17, no. 1, pp. 227–238, 1996.
  * [14] Y. Zhang, J. Goh, L. L. Win, and V. L. Thing, “Image region forgery detection: A deep learning approach.” _SG-CRC_ , vol. 2016, pp. 1–11, 2016.
  * [15] D. Tralic, I. Zupancic, S. Grgic, and M. Grgic, “Comofod—new database for copy-move forgery detection,” in _Proceedings ELMAR-2013_. IEEE, 2013, pp. 49–54.
  * [16] Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, and D. Ren, “Distance-iou loss: Faster and better learning for bounding box regression,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 07, 2020, pp. 12 993–13 000.
  * [17] G. Mahfoudi, B. Tajini, F. Retraint, F. Morain-Nicolier, and M. Pic, “Defacto: Image and face manipulation dataset,” in _2019 27th European Signal Processing Conference (EUSIPCO)_ , 2019.
  * [18] Q. Yang, D. Chen, Z. Tan, Q. Liu, Q. Chu, J. Bao, L. Yuan, G. Hua, and N. Yu, “HQ-50K: A Large-scale, High-quality Dataset for Image Restoration,” Jun. 2023, arXiv:2306.05390 [cs]. [Online]. Available: http://arxiv.org/abs/2306.05390
  * [19] ——, “HQ-50K: A Large-scale, High-quality Dataset for Image Restoration,” Jun. 2023, arXiv:2306.05390 [cs]. [Online]. Available: http://arxiv.org/abs/2306.05390