# Sanitization of Visual Multimedia Content: A Survey of Techniques, Attacks,
and Future Directions

Hanaa Abbas<EMAIL_ADDRESS>0000-0001-7577-8966 and  Roberto Di Pietro
<EMAIL_ADDRESS>0000-0003-1909-0336 College of Science and Engineering,
Hamad Bin Khalifa UniversityDohaQatar

(2022)

###### Abstract.

The exploding rate of multimedia publishing in our networked society has
magnified the risk of sensitive information leakage and misuse, pushing the
need to secure data against possible exposure. Data sanitization—the process
of obfuscating or removing sensitive content related to the data—helps to
mitigate the severe impact of potential security and privacy risks.

This paper presents a review of the mechanisms designed for protecting digital
visual contents (i.e., images and videos), the attacks against the cited
mechanisms, and possible countermeasures. The provided thorough
systematization, alongside the discussed challenges and research directions,
can pave the way to new research.

multimedia sanitization, multimodal de-identification, visual obfuscation,
anonymization, security, privacy

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††journal:
CSUR††journalvolume: 0††journalnumber: 0††article: 0††publicationmonth:
1††ccs: Security and privacy Pseudonymity, anonymity and untraceability††ccs:
Security and privacy Privacy-preserving protocols††ccs: General and reference
Surveys and overviews

##  1\. Introduction

Over the recent years, the volume of images and videos generated, shared, and
stored by social media platforms, sensor and IoT networks, and video
surveillance systems, to cite a few, has drastically increased. However, the
exploding amount of available data has increased the risk of privacy breaches.
This risk is exacerbated by the availability of affordable hand-held
electronic devices (e.g., smartphones, cameras) and high-speed 5G networks,
that have made it much easier for the average user to create and globally
share video and image files. Such visual content is often uploaded to
untrusted third parties or cloud providers for storage and further processing.
Therefore, the need for privacy-preserving technologies is more critical now
than ever.

Privacy is a serious concern in our networked societies, and thus several data
protection laws and international standards were enacted worldwide. Similarly,
most social media networks enforce policies to combat the dire consequences of
privacy violations on users, such as social impact (e.g., damaged reputation,
identity theft), financial losses, and even physical harm. For over two
decades, considerable research was directed towards privacy-enhancing
approaches for digital media. The proposed solutions can be categorized into
two branches (Padilla-López et al., 2015): (a) access restriction; and, (b)
controlling media content by modifying the sensitive parts (i.e., multimedia
data sanitization). In this survey, we consider only the second approach. That
is, we do not include works based on hardware-oriented solutions (including
intervention and blind vision (Sah et al., 2017)), access control (e.g., user
authentication and roles management), secure processing (e.g., secure
multiparty computation and naïve encryption), and data hiding (e.g.,
steganography and digital watermarking).

Data sanitization includes the obfuscation or removal of sensitive information
such as personally identifiable information (PII) to preserve the user’s
privacy. In some use-cases, some information leakage is needed in the
obfuscated media content to maintain its usability and intelligibility; hence,
a degree of privacy loss is always inevitable. Throughout the paper, this
concept is highlighted as the privacy-utility trade-off. Nowadays, research
efforts are focused on attaining provable security guarantees that precisely
characterize the degree of information leakage about the sanitized multimedia.
In the past, the sanitization process was performed manually by trained
experts. However, in recent years, with the advent of Machine Learning
(ML)-based techniques, there is a widespread trend to semi or fully automate
the cumbersome and costly process. However, sanitization is not the only
process being automated; attacks against it are also automated using state-of-
the-art techniques, such as neural networks, face recognition (Zhao et al.,
2003), etc.. Consequently, in this survey, we additionally review the
countermeasures devised to thwart such automated attacks.

Data sanitization technologies are a double-edged sword. Indeed, we note that
many of the to-be-discussed obfuscation techniques can also have malicious use
cases. For example, adversaries may utilize obfuscation methods to evade being
recognized by biometric verification and video surveillance systems. However,
since obfuscation methods should withstand conventional recognition and
identification algorithms that the adversary may use, we do not differentiate
obfuscation based on its intent. Typically, in data sanitization, the region
hosting sensitive information is either distorted, replaced, or removed. On
the one hand, distortion modifies the original object to render it
unintelligible, while replacement substitutes the element with a visually
similar one. Object removal ensures complete protection; however, the utility
is severely compromised. On the other hand, distortion and replacement methods
may leak information about the obfuscated content while achieving higher
utility than removal. Fig. 1 depicts the privacy-utility trade-off between the
obfuscation categories. The figure also illustrates the possibility of
combining the different methods, a technique called multimodal obfuscation.

Figure 1. The Utility-Privacy Trade-off of Obfuscation Methods

A chart of three intersecting circles; the top circle represents removal, and
the bottom two represent distortion (left) and replacement (right). A
downwards arrow shown on the left for utility, and an upwards arrow shown on
the right for privacy.

As it will be shown later in the paper, there is no single best approach to
sanitization; each redaction solution comes with its benefits and drawbacks.
Therefore, one of the objectives of this contribution is to help
practitioners, in industry and academia, in devising the optimal combination
of techniques to reach a certain degree of privacy, while maintaining an
acceptable value of the utility function.

Over the past years, not many attempts were made to survey the literature on
sensitive information protection using obfuscation methods. In one of the
first surveys in the area, Padilla-López et al. (2015) summarized hardware and
software solutions for image and video redaction in the context of privacy-
aware intelligent monitoring systems (Padilla-López et al., 2015). The authors
classified protection methods into five large categories: intervention, blind
vision, secure processing, redaction, and data hiding. Intervention and blind
vision are mainly hardware-based solutions. In contrast, in our survey, we
limit our focus to methods concerning direct modification of the multimedia’s
regions of interest (ROI) to obscure information (i.e., redaction). A
comprehensive survey of de-identification in multimedia content for protecting
privacy is given in (Ribaric et al., 2016). Ribaric et al. (2016) (Ribaric et
al., 2016) presented de-identification of non-biometric, physiological, and
behavioral biometric identifiers and soft-biometric identifiers found in
images and video. In (Sah et al., 2017), Sah et al. (2017) focused on video
redaction, including the problem of object detection and tracking. In
(Rakhmawati et al., 2018), Rakhmawati et al. (2018) presented a very brief
survey about the techniques of image privacy protection in a five-page
conference paper. Their paper describes the characteristics, evaluation
methods, and privacy protection methods that apply to images. The surveys in
(Winkler and Rinner, 2014) and (Liu et al., 2020) address two specific
applications of image privacy protection in visual sensor networks and image
sharing on online social networks, respectively. In comparison to the cited
surveys, we include a more detailed discussion of the observed current
research trends and possible future research directions. We also provide a
general discussion in each multimedia obfuscation section reviewing the
provided methods, reference applications, key properties, and challenges.
Additionally, our survey is the first comprehensive survey that addresses
attacks and countermeasures concerns of visual media obfuscation in one paper.

Table 1 compares the main existing surveys in the literature (including our
contribution) in terms of the addressed media type (images and video) and
whether attacks and countermeasures are explored. As it can be seen from the
table, other surveys go back in time and are not as comprehensive as ours.

Table 1. Comparison of Main Surveys on Multimedia Security and Privacy

Survey & Year | Digital Media | Attacks & Solutions | Future Directions
---|---|---|---
Images | Video
2014 (Winkler and Rinner, 2014) | ✓ | $\times$ | $\times$ | $\times$
2015 (Padilla-López et al., 2015) | ✓ | ✓ | $\times$ | $\times$
2016 (Ribaric et al., 2016) | ✓ | ✓ | $\times$ | $\times$
2017 (Sah et al., 2017) | ✓ | ✓ | $\times$ | $\times$
2018 (Rakhmawati et al., 2018) | ✓ | $\times$ | $\times$ | $\times$
2020 (Liu et al., 2020) | ✓ | $\times$ | $\times$ | $\times$
Our survey | ✓ | ✓ | ✓ | ✓

Contributions. In this paper, we provide several contributions: We start with
a thorough systematization of sanitization techniques for images and video; we
later review the attacks that these techniques are subject to; in addition, we
discuss current countermeasures; and, finally, we highlight further research
directions. In detail, our main contributions are summarized as follows:

  * •

Categorization of the obfuscation methods according to their underlying
mechanism (distortion, replacement, or removal) and highlighting of the
properties expected to be preserved after obfuscation.

  * •

Summary of the most recent and relevant contributions in the literature about
the obfuscation techniques for each media type and their known attacks and
possible countermeasures.

  * •

Provisioning of an in-depth discussion for each media type exploring topics
such as obfuscation challenges, current trends, and future research
directions.

Roadmap. The remainder of the paper (see paper overview in Fig. 2) is
organized as follows. We detail the sanitization techniques used for images
(Section 2) and video (Section 3), where each section also provides a
comprehensive discussion about the described techniques in terms of their key
properties, the attacks they are subjected to, and possible research
directions. We establish our conclusions in Section 4.

Figure 2. Overview of the surveyed domains, techniques, attacks, and provided
discussion.

A mind map representing the paper overview.

##  2\. Image Obfuscation

The number of images captured daily is rapidly increasing due to the
availability and affordability of high-quality cameras embedded in personal
devices and surveillance systems, among others. A vast percentage of these
images are uploaded to online social networks or cloud-based platforms for
processing and storage. To put numbers into perspective, in November 2020,
Google Photos announced that it had stored more than 4 trillion images
111Google Blog: https://blog.google/products/photos/storage-changes/. Every
day, approximately 300 million photos are uploaded to Facebook and 95 million
to Instagram 222https://dustinstout.com/social-media-statistics/. Thus, the
critical regions in the images must be identified then removed or replaced to
minimize the risk of disclosing sensitive or personal information.

An automated sanitization system typically involves multiple steps. The
desired security level must be clearly defined in the pre-processing stage,
including which sensitive objects must be identified and protected. The second
stage involves two further steps: the identification step and the obfuscation
step. In the identification step, the region of interest (ROI) is detected
using well-established image recognition techniques (e.g., face recognition).
The boundaries of the ROI are then localized (i.e., image segmentation) and
annotated. An obfuscated image is produced by applying an obfuscation
algorithm on the annotated ROI. In the post-processing stage, the output image
is evaluated for its security based on the pre-defined security goals, i.e.,
whether it should withstand human recognition, machine recognition, or both.
Fig. 3 depicts the main integral components and steps of an image sanitization
system.

Figure 3. Major Steps and Components of an Image Obfuscation System

The process of image obfuscation; input image data go into a pre-processing
block, followed by processing. The resulting output is sanitized image data
which go into the post-processing block.

The authors in (Ribaric et al., 2016) categorized sensitive image attributes
as biometric, non-biometric, and soft-biometric. This paper follows the same
taxonomy; however, we extend it to include confidential and censored
categories to incorporate attributes that are not personal but must be
secured. The types and examples of sensitive content in images are listed in
Table 2. Most images have multiple types of objects that need to be identified
and protected, known as the multimodal identification problem. For instance,
even if the face is obfuscated in an image, soft biometrics such as body
silhouette, clothing style, and skin color can be strong leads to a person’s
identity. Even after the obfuscation process, some residual visual features
will remain. They are either left intentionally to maintain the utility and
intelligibility of the image or missed unintentionally due to detection
errors. This is repeatedly referred to as the privacy-utility trade-off in the
literature. According to the application scenario, the obfuscated image may be
required to retain specific characteristics: (a) reversibility, (b) utility,
and (c) naturalness. Reversibility is a property achieved by the provision of
additional translational variables. These variables are expected to remain
secret or partially unknown to an attacker; otherwise, the obfuscation
security is compromised. The utility and naturalness of images are highly
desirable when images are intended for public use or model training for
automatic recognition and classification algorithms. Also, maintaining a
natural appearance helps conceal the fact that some identifiers were
obfuscated, which can be a security goal on its own. Besides the usability and
intelligibility trade-off, another common challenge is providing provable and
quantifiable privacy or security guarantees. This section explores the
different obfuscation techniques and the possible attacks an adversary may
perform to infer the hidden content. Furthermore, we suggest some
countermeasures and insights regarding current trends in the research and
potential areas of improvement.

Table 2. Types of Sensitive Content in Images

Type |  Description |  Examples
---|---|---
Biometric |  Unique, measurable and permanent personal identifiers (Ribaric et al., 2016). |  Face, iris, ear, fingerprint
Soft biometric |  Vague physical, behavioral or adhered personal characteristics that are not necessarily unique or permanent (Ribaric et al., 2016). |  Height, eye color, body shape, age, gender, skin color, tattoos, birthmarks, scars
Non-biometric |  Personal identifiers that provide non-physical and non-behavioral contextual information about the individual. Such attributes are either temporary and changeable. |  Clothing, hairstyle, location, license plate, identification cards (e.g. health card, driver’s license), credit cards
Confidential |  Non-personal attributes that should be withheld from public for security and privacy reasons. |  QR codes, cheques, keys, classified images (e.g., high-security buildings)
Censored |  Visual censorship of content due to laws and regulations. |  Branded products, pornography, gruesome content

###  2.1. Techniques

We propose the following classification of image obfuscation techniques based
on the primary mechanism of obfuscation (distortion, replacement, or removal
of image features). The list of seven categories is presented in Table 3. The
categories are image transformation, encryption-based protection, adversarial
perturbation, image de-identification, abstraction, masking and blocking, and
inpainting methods. In the following subsections, we provide explanations and
further discussions about each technique. In Table 3, we rate the main
obfuscation techniques based on the protection level against human and machine
attackers separately, the utility of the obfuscated image, and the ability to
reverse the obfuscation. A low utility rating indicates severe quality
degradation and an absence of naturalness and intelligibility. A medium rating
indicates a general lack of naturalness or intelligibility. A high rating
indicates that naturalness, usability, and quality are preserved. In terms of
security, a low rating indicates that the obfuscated image is easily
detectable and recognizable either with the naked eye or using image detection
and recognition algorithms. A medium rating indicates the existence of known
attacks that can leak information; however, the attacks do not have high
success guarantees. A high rating indicates that the obfuscated image has a
low risk of information disclosure.

Table 3. Main Image Protection Techniques in the Literature (L: Low, M:
Medium, H:High)

Category | Main Techniques | Subtype | Sub-Subtype | Security Level | Utility | Reversibility
---|---|---|---|---|---|---
Human | Machine
Distortion | Image Transformations | Image filtering | Pixelization | L | L | L | ✓ (lossy)
Blurring
Edge
Compression
Image warping | - | L | L | L | ✓ (lossy)
Aesthetic transforms | Cartooning | L | L | M | ✓ (lossy)
False color
Style transfer
Encryption-based Protection | Partial Encryption | P3, TPE | H | H | L | ✓ (lossless)
Scrambling | -
Adversarial Perturbation | - | - | L | M | H | $\times$
Replacement | Image De-identification | Face synthesis |  $k$-Same | H | H | H | $\times$
ML-based
Face morphing | - | H | M | M | $\times$
Abstraction | - | - | H | H | M | $\times$
Removal | Blocking/Masking | - | - | H | H | L | $\times$
Inpainting | - | - | H | H | M | $\times$

####  2.1.1. Image Transformations

Image transformation is an image editing method that refers to mapping an
input image to a processed output image using a function. Early attempts at
redacting photos relied on image transforms to hide sensitive regions. Image
transformation techniques, specifically pixelization and blurring, are
typically used for censorship in television and online news (Rakhmawati et
al., 2018). Here we consider image filtering, image warping, and aesthetic
transforms as types of image transformations used for distorting the
appearance of the original image (see Table 4 for examples). In image
filtering and image warping, the obfuscated image is a degraded version,
characterized by its information and utility loss. Since the image transforms
are based on a parameterized model, this category of obfuscation is highly
vulnerable to ML-assisted attacks (McPherson et al., 2016; Tekli et al.,
2019). Such attacks are trained to reconstruct an approximated (i.e., lossy)
original from the redacted image or deduce the image’s original using evident
correlations from within a set of unobscured images. As per aesthetic
transformations, they modify the input image to produce an abstract or a
cartoon equivalent which may thwart weak recognition systems. Overall, these
methods are ineffective against human and machine attackers.

Table 4. Main Image Transformation (Editing) Techniques in the Literature

Category | Technique | Example |  Reference & Year
---|---|---|---
Image filtering | Pixelization |  |  2018 (Fan, 2018), 2019 (Fan, 2019)
| Blurring |  |  2016 (Fan, 2018)
| Edge Detection |  |  2019 (Hasan et al., 2019)
Image warping | e.g. Twirl effect |  |  2013 (Korshunov and Ebrahimi, 2013b)
Aesthetic Transforms | False Color |  |  2017 (Çiftçi et al., 2017)
| Cartooning |  |  2014 (Erdélyi et al., 2014), 2017 (Hasan et al., 2017), 2018 (Larson et al., 2018)

Image filtering: Image filtering has many applications in image processing,
including edge detection, smoothing, sharpening, and noise reduction. The
image is treated as an $M$ x $N$ matrix. Each cell in the matrix contains an
integer from 0 to 255 (0 is black and 255 is white). An image can also have
multiple channels such as RGB (red-green-blue), HSV (hue-saturation-value) and
YUV (Yu et al., 2021). Filtering is a mathematical operation where a matrix
(aka, kernel) is convoluted with each pixel value and its neighbors within an
image.

Pixelization. Pixelization (aka, mosaicing) is a widely adopted technique
where the region of interest (ROI) is divided into a square grid (e.g.,
8$\times$8, 16$\times$16) or a rectangular grid. The average color value of
the pixels within each box is computed. Then, the value of each pixel within
the box is replaced with the computed average. Pixelization can be considered
a reduction in the resolution of the obfuscated region (McPherson et al.,
2016). For a pixel box of size $n\times n$, the resolution is effectively
reduced by a factor of $n^{2}$ (McPherson et al., 2016). Increasing the size
of the pixel box increases the level of security as more pixels are averaged
together. As can be seen in Fig. 4, pixelated images of smaller pixel boxes,
e.g., 5$\times$5, are still human-recognizable.

(a) Original

(b) 5$\times$5

(c) 10$\times$10

(d) 20$\times$20

(e) 30$\times$30

Figure 4. Pixelization: 5$\times$5, 10$\times$10, 20$\times$20, and
30$\times$30 grids.

Four pixelated versions of Tux the penguin (Linux logo).

In (Fan, 2018), Liyue Fan (2018) presented an image pixelization method that
extends the notion of differential privacy, providing rigorous privacy
guarantees. Differential privacy (Dwork, 2008) (aka, statistical disclosure
control) is the state-of-the-art privacy paradigm for statistical database
sanitization. It defines a parameter $\epsilon$ that corresponds to the degree
of privacy, where the smaller the value of $\epsilon$, the stronger the
privacy. The algorithm first performs pixelization on an input image and
applies Laplace perturbation to the pixelized image. The Laplace perturbation
uses the notion of $m$-neighborhood, which protects up to $m$ pixels. The
output image would closely resemble a normally pixelized image except for $m$
grid cells. The algorithm guarantees the indistinguishability of the output of
two neighboring images having the same dimensions and differing by at most $m$
pixels. As $m$ increases, the privacy is increased, but the utility is lowered
as the Laplace perturbation noise increases. Fig. 5 shows a pixelized image
(left) and its differentially private (DP) alternative (right), where the
pixel box size is $16\times 16$, $m=16$, and $\epsilon=0.5$. The noise added
after pixelization protects the image from machine-based recognizers and
provides provable and quantifiable privacy guarantees. The author also
extended the concept of differential privacy to blurring. The algorithm first
adds Laplace noise to each pixel and then applies Gaussian blur to smoothen
the output.

Figure 5. Pixelized (left) and DP-pixelized (right) images for $16\times 16$
pixel grid, $m=16$, $\epsilon=0.5$ (Fan, 2018).

Two grayscale pixelated images, where the pixel grids of second image on the
right appear in different positions than the first image on the left.

In (Fan, 2019), the same author (2019) expanded on DP pixelization to yield
higher utility. The output of this method exhibits more resemblance to the
original image. Compared to standard pixelization, the accuracy of re-
identification using CNN-based attacks is lowered from 96.25% to 82.5% for
$\epsilon$ = 0.5 and 17.5% for $\epsilon$ = 0.1. The author used a generalized
notion of DP known as metric privacy. Metric privacy guarantees a level of
indistinguishability proportional to the distance between two inputs. Meaning,
the mechanism’s output is roughly the same for all visually similar images in
a set, and an adversary cannot infer the exact input image by observing the
obfuscated image. The obfuscation algorithm involves two major steps:
transformation and random sampling. The first step transforms a sensitive ROI
into a k-dimensional feature vector using Singular Value Decomposition (SVD).
SVD is used as a (invertible) perceptual image hashing method that captures
geometric features and characteristics of the image data. The higher the value
of k, the higher the resemblance between the input and obfuscated ROI. Next,
the vector is randomly sampled according to a certain probability
distribution, satisfying metric privacy. The sampled vector is processed with
an inverse transform to produce the obfuscated ROI. The technique was
evaluated using Mean Square Error (MSE) and Structural Similarity (SSIM)
metrics between the obfuscated ROI and the original ROI. SSIM is a value
between 0 and 1 representing the holistic similarity between two images, where
1 indicates identical images (Tekli et al., 2019).

Blurring. Blurring is another image quality degradation technique. There are
several types of blur filters. A Gaussian blur is where pixels within the ROI
are convoluted with a Gaussian kernel to achieve a “smoothing” effect. The
blurriness level is controlled by the standard deviation $\sigma$ parameter.
Another type of blurring is motion blur. The motion blur alters the details of
an image by replicating the effect of a synthetic camera’s motion. The degree
of blurriness is affected by the length and the angle of the synthesized
motion. Box blur, also known as a linear box filter, is a form of a low-pass
filter where each pixel is replaced with the average value of its neighboring
pixels.

Edge Detection. In image processing and computer vision, edge detection is a
fundamental technique for feature detection and feature extraction. It
includes various mathematical methods that identify the points at which the
image brightness discontinues or changes sharply. Image edges preserve the
shape and some internal details of the ROI (Hasan et al., 2018). Unlike
pixelization and blurring, edge detection is not a typical obfuscation method
and it is considered a form of abstraction. Nonetheless, there were a few
mentions of edge detection as an obfuscation method in (Winkler and Rinner,
2014; Hasan et al., 2019, 2018). The level of obfuscation is controlled by an
“edge filter” parameter which affects the strength and visibility of the edges
and the inner details of the image.

Discussion. The methods detailed in this section are regarded as naïve and
conventional methods that are no longer secure against machine detectors and
that can still leak information to human observers. In (McPherson et al.,
2016), pixelated images of objects belonging to 1 out of 10 classes and
smaller pixel boxes, e.g., 4$\times$4, were easily recoverable using neural
networks with an identification accuracy of over 70%. With 16$\times$16 box
size, the accuracy of recovery is about 31%. In (Li et al., 2017) and
(Vishwamitra et al., 2017), the authors investigated the effectiveness of
pixelization and blurring against human recognition as well and explored its
impact on user satisfaction. The evaluation proved that both methods are
ineffective against human recognition. However, considering user satisfaction,
they performed better than blocking or masking methods but still were not
favorable.

There are other image processing techniques that are sparsely cited as forms
of visual privacy protection such as scaling, stretching, rotation, modifying
saturation and brightness, grayscaling, and lossy compression (Poller et al.,
2012). However, they are no longer viable due to the robustness of image
fingerprinting and image recognition methods that can easily identify matching
images even after severe quality degradation. Lossy compression (aka, high
compression or downsampling) is an irreversible data encoding that
significantly decreases the peak signal-to-noise ratio (PSNR) (Poller et al.,
2012; Kim and Lee, 2014) of the image and produces inexact approximation or
partial discarding of the image data. It is commonly used to reduce data size
during storage and transmission. The most widely used lossy compression scheme
applies discrete cosine transform (DCT) on the images. Existing compression
standards are JPEG and JPEG-2000 for images, and M-JPEG, M-JPEG 2000, MPEG-4,
or AVC/H.264 for videos (Kim and Lee, 2014). Similar to pixelization and
blurring, the utility of the image is lowered after compression; however, it
can remain easily recognizable by the end-user. Research has shown that
compression is also vulnerable to super-resolution ML-trained models (Poyser
et al., 2020; Wang et al., 2020; Anwar et al., 2020).

Image warping: Image warping is a geometric transformation of an image such
that any depicted object is significantly distorted. Warping destroys the
details and relationship between neighboring pixels while retaining the
general shapes of the image. It is commonly used for animation and artistic
purposes. In (Korshunov and Ebrahimi, 2013b), Korshunov et al. (2013) proposed
an algorithm that uses warping techniques to obfuscate faces. The algorithm
works as follows: the face region is detected using face recognition
algorithms. Then, a set of key pixels in the image are selected from the eyes,
nose, mouth, and sides of the face. These pixels are shifted (i.e., their
coordinates changed) by a random shifting distance that is decided using a
pseudo-random algorithm. The next step is to estimate the transformation
matrix based on the original and destination coordinates of the shifted
pixels. The transformation is applied to the remaining pixels using the
estimated matrix, while any gaps are interpolated using a bicubic algorithm.
The discussed process is reversible if the inverse transformation matrix can
be re-estimated (Cavedon et al., 2011). However, due to the interpolation of
some pixels, an unwarped image is only an approximation of the original. The
strength of the warping effect, and thus the privacy, is based on the shifting
distance. Ideally, a warped face would be detectable by face detection
algorithms but unidentifiable by face recognition algorithms. Security is
enforced by using a secret key for seeding the pseudo-random algorithm and
encrypting the selected key pixels.

Aesthetic Transforms: Artistic transformations on images are mainly intended
for improving the utility of obscured images. It can be beneficial for online
social networks, where sensitive information can be hidden without degrading
the user experience. In (Hasan et al., 2019) and (Hasan et al., 2018), the
authors conducted an objective evaluation to study aesthetic transforms as a
visual obfuscation method and their impact on viewer satisfaction. The study
applied cartooning and false colors to conventional obfuscation methods
(pixelization, edge detection, and masking) known for their high utility loss
and low visual perception. Despite the applied beautifications, the results
show that user satisfaction did not significantly increase.

False Color. False colors (see examples in Fig. 6) are typically used as a
visualization aid in image processing. They are used to depict invisible
features that our eyes cannot normally see (e.g., invisible spectral emissions
and sensitivities, line of sight analysis). An RGB input image is first
transformed into grayscale. Then, the grayscale value is mapped to an RGB
value based on a predefined color palette. The mapped RGB value then replaces
the original pixel value. The primary advantages of this technique are that
the intelligibility of the image is not compromised, the selection of an ROI
is not required, and it can be used in conjunction with other obfuscation
techniques (Çiftçi et al., 2017). However, false colorization is partially
reversible (i.e., lossy) because: (a) the color palette may not be one-to-one;
and, (b) the initial RGB to grayscale conversion is a lossy operation—the
color information is not retained. Çiftçi et al. (2017) (Çiftçi et al., 2017)
presented a reversible false color-based obfuscation for the JPEG standard,
where the entire image is colorized, and the restorative information is
encrypted and embedded in the file’s metadata. The output of the system is the
false-color image (FI), whose JPEG APP Markers contain the encrypted and
compressed values of the difference image (DI), sign image (SI), the
histograms of the original image, and the color palette. To ensure the
reversibility of the scheme, instead of using direct RGB-to-gray conversion, a
false-color value is computed for each color value in the original image.
Since most color palettes are not one-to-one, a pseudo-inverse image is
computed using the inverse table lookup. The resulting FI is then encoded to
the JPEG standard. DI is obtained by subtracting FI from the pseudo-inverse of
the original image. SI is included since the difference can be negative.

Figure 6. False color-based protection using three different palettes (Çiftçi
et al., 2017).

Three versions of false colorization using palettes: blue, green-pink, and
hardcandy.

Cartooning. Cartooning achieves an effect similar to watercolor painting,
where an ROI is converted into an aesthetically pleasing abstracted frame. For
images containing faces, cartooning maintains a high utility since it
preserves general properties like behavioral information, gender, emotion, and
at the same time obscures facial identity (Erdélyi et al., 2014; Hasan et al.,
2017). In (Erdélyi et al., 2014), Erdelyi et al. (2014) proposed an adaptive
mechanism where the intensity of the cartooning filter can be changed
according to the level of protection and utility required for specific scenes.
The method uses two key techniques: color filtering and edge enhancements,
i.e., smoothing the areas with moderate color variations to single-colored
areas (Erdélyi et al., 2014). Their evaluation metrics were the structural
similarity (SSIM) index and peak signal-to-noise ratio (PSNR). The results
show that cartooning achieves higher privacy and utility than blurring and
pixelization.

Hasan et al. (2017) in (Hasan et al., 2017) proposed a cartooning algorithm
having two major components: the first is cartooning the entire scene, and the
second is using computer vision to detect the sensitive objects and replace
them with visually similar and randomly selected clip-art images of the same
class. Their method is intended to preserve semantic scene information while
obfuscating fine-grained details. The authors note several challenges in the
accuracy of identification and localization of objects, mainly: the selection
of suitable clip-art, and the alignment of the clip art with the removed
object in terms of scale, position, and orientation in an aesthetically
pleasing manner. The results showed a general acceptance by the participants
of the images’ semantics, aesthetics, and privacy. However, the major downside
is that the replacing clip-art is selected randomly from a dataset and,
therefore, some participants were not pleased with the quality of the clip
arts.

ML-based algorithms are being utilized for cartooning and aesthetic style
transfers. For example, in (Larson et al., 2018), Larson et al. (2018) applied
to the entire image a cartooning model based on Generative Adversarial Network
(GAN). They intended to confuse image classifiers not to predict the correct
category of the image while maintaining its visual aesthetics. Aesthetic style
transfer (illustrated in Fig. 7) can be achieved using Convolutional Neural
Networks (CNN), where a model is trained to generate an output combining the
content of an image with the style of another (Li and Lin, 2019).

Figure 7. Examples of aesthetic CNN-based style transfer for face obfuscation
(Li and Lin, 2019).

Three examples of abstraction of an image of Donald Trump.

####  2.1.2. Encryption-based Protection

A proper encryption of the photos (e.g., using AES in CBC mode) provides
complete security. Still, it destroys the information that is not privacy-
breaching, therefore, hindering the ability to perform any image processing
task without the secret key. Several technologies solve this problem by
providing what is referred to as conditional access (Padilla-López et al.,
2015) and significantly distorting only the sensitive region of interest in
the image while leaving the rest as is. The main advantage of this type of
protection is that the original data can be perfectly recovered using the
secret key. The drawbacks are that a secure channel to exchange the key and
high computing power are needed, making it inefficient for real-time
applications. Encryption and scrambling can be performed in different image
domains: pixels, bitstream, or Discrete Cosine Transform (DCT) coefficients
(Yuan and Ebrahimi, 2017).

P3. Ra et al. (2013) (Ra et al., 2013) proposed P3 encryption for JPEG-
compliant images, which splits the image, based on an AC coefficient
threshold, into a JPEG public part and a private part. The two parts are
shared independently, and a secret key is needed to uncover the private part.
Their combined size does not significantly exceed the size of the original
image. P3 assumes that most information about an image is carried in the 0th
DC coefficient and the remaining 63 AC coefficients. Therefore, P3 encrypts
the most significant bits of the significant DCT coefficients in the private
part; the remaining coefficients and least significant bits are left in the
public part. The image can be reconstructed by decrypting the private parts
and positioning them back in the public part. According to the authors, P3 was
explicitly designed to protect against automated recognition technologies
since the public image does not resemble the original image (Ra et al., 2013).
Later on, it was uncovered that P3 is not effective in privacy preservation
against artificial neural networks (McPherson et al., 2016; Fan, 2018). When
the splitting threshold is not small enough, some visual information could be
leaked from the public image. For a threshold of 1, the highest privacy
setting of P3, Mcpherson et al. (2016) (McPherson et al., 2016) achieved an
83% re-identification accuracy of obfuscated faces using a simple neural
network, and 97% for a threshold of 20.

JPEG Metadata Embedding. Several works in the literature leveraged the
metadata (application segments or APPn marker) in the JPEG standard to fully
reverse the obfuscation. In (Yuan et al., 2015a), an JPEG sub-image is
constructed from the DCT coefficients of the original image, where the
coefficients outside of the identified ROI are set to zero. A mask matrix is
built from the shape, size, and position parameters of the obfuscated ROI. The
sub-image is encrypted using symmetric encryption. The mask matrix and other
metadata are also encrypted and embedded in the image file. A similar approach
in (Farrugia, 2014) proposed a two-level reversible watermarking scheme, used
in conjunction with any obfuscation process, where the residual information
needed to reverse the obfuscation process is compressed, authenticated,
encrypted, and finally embedded within the obfuscated image. The authors
implemented their scheme with $k$-same face replacement, where the reversing
information is the difference between the original and obfuscated image. Other
JPEG-specific partial encryption methods include Cryptagram (Tierney et al.,
2013) and Thumbnail Preserving Encryption (TPE) (Wright et al., 2015). TPE
scheme allows the reconstruction of a low-resolution thumbnail from an
encrypted image while preventing the extraction of any further data.

Scrambling. Image scrambling (or image permutation) is the process of
rearranging pixel tiles randomly in the spatial domain to break the
correlation between the neighboring pixels and make the image unintelligible
(Poller et al., 2012; Yuan et al., 2015b; Mondal, 2018). It can also be
performed in DCT domain or codestream domain depending on the media format
(Rakhmawati et al., 2018). For a useful and comparative review of the various
scrambling techniques proposed in the literature, refer to the survey given in
(Mondal et al., 2017) by Mondal et al. (2017).

Permuting DCT coefficients or image tiles is insufficient to achieve secure
scrambling. Scrambling can be solved through a brute force attack whose
complexity is $t!$ for $t$ tiles (Sharma and Chen, 2018). An adversary can
further reduce the attack’s complexity by analyzing the boundary, color, shape
of the content, and texture of neighboring blocks in the image. Additionally,
scrambling is impacted by lossy compression (Korshunov and Ebrahimi, 2013b;
Poller et al., 2012). When scrambling a compressed image, a blurring effect
appears at the borders of the permuted pixel grids. Fig. 8 shows an example of
a spatial-domain permutation and de-permutation on a compressed JPEG image. In
(Poller et al., 2012), Poller et al. (2012) proposed a secure jigsaw
scrambling method that works with compressed JPEG images. The border blurring
effect is reduced by replacing the pixel values at each grid’s margins with
the pixel values at the border during de-scrambling. To achieve higher
security, the authors increased the number of possible permutations by
shifting, rotating, and mirroring pixel grids randomly based on a secret key,
and then by pseudo-randomly modulating the pixels’ channel intensity (Poller
et al., 2012). The authors suggested salting the encryption key for additional
security and decreasing the grid size to increase the number of permutations.
Sharma et al. (2018) (Sharma and Chen, 2018) applied block-wise Randomized
Multidimensional Transformations (RMT) to the pixel matrix of the image.
Acting like the encryption key, a random orthogonal (i.e., rotation) or a
projection matrix multiples the image’s matrix. Then, random additive noise
uniformly selected from a known range is added to the multiplication output.

Figure 8. Conventional Spatial-Domain Permutations (Poller et al., 2012):
(left: original, center: permuted, right: de-permuted with blurred grid
borders).

An example of scrambling, where the left image is split into pixel tiles which
are shuffled around randomly to produce the center image.

####  2.1.3. Image De-identification

De-identification is the process of hiding, eliminating, or substituting
personal identifiers (aka, personally identifiable information or PII) with
surrogate identifiers to prevent the disclosure and use of private data for
unintended and potentially harmful purposes. It aims to hide personal
information from human recognizers, automatic recognition techniques, or both.
The terms anonymization and de-identification are often used interchangeably
in the literature. Nonetheless, the terms can be differentiated based on
whether the obfuscation process is reversible (de-identification) or
irreversible (anonymization) (Ribaric et al., 2016). Table 5 lists the main
attributes and features (biometric, soft-biometric, and non-biometric) that
can be extracted from facial and body landmarks in images. The required level
of protection and utility determines which of the listed features to obfuscate
and which ones to preserve. In Table 5, we refer to works in the literature
investigating recognition and de-identification (and, in some cases, the
preservation) of the listed features.

The human face is the most prevalent biometric identifier in photos (Ribaric
and Pavesic, 2015; Meden et al., 2018). A trivial solution to de-identifying
faces involves blacking out the eyes, mouth, or the entire face, where the
first two fail to thwart currently robust face recognition solutions (Newton
et al., 2005). Nowadays, most research focuses on achieving face obfuscation
that is photo-realistic while maintaining standard facial characteristics. In
this section, we look at image de-identification from the perspective of face
de-identification. Nonetheless, image de-identification also includes the
obfuscation of non-biometric identifiers such as license plates (McPherson et
al., 2016), QR codes (Zhang et al., 2019), etc..

Table 5. Image De-Identification: Balancing Privacy-Utility in Facial & Body
Features

| Feature | Type | Reference
---|---|---|---
Body | Fingerprint | Biometric | (Yoon et al., 2012; Lugini et al., 2014)
Skin tone | Semi-biometric | (Du et al., 2014; Qin et al., 2017; Karras et al., 2019)
| Tattoos, Scars,
---
Birthmarks
Semi-biometric | (Marčetic et al., 2014; Hrkać et al., 2016)
Clothing style | Non-biometric | (Brkić et al., 2017; Brkic et al., 2017; Prinosil, 2018)
Body Pose, Activity | Semi-biometric | (Ren et al., 2018; Nambiar et al., 2019)
Face | Iris, eyes | Biometric | (Thavalengal et al., 2014; Zhang et al., 2018)
Ear | Biometric | (Emeršič et al., 2017; Jiddah and Yurtkan, 2018)
Gender | Semi-biometric | (Du et al., 2014; Gross et al., 2005; Othman and Ross, 2014)
Age | Semi-biometric | (Du et al., 2014)
Hair style, hair color | Non-biometric | (Qin et al., 2017; Brkić et al., 2017; Prinosil et al., 2015)
Facial Expression | Semi-biometric | (Gross et al., 2005)

Face De-identification: Table 6 shows a breakdown of the reviewed solutions
for face de-identification: face morphing and face synthesis using $k$-same
and adversarial learning methods.

Table 6. The Main Methods of Face De-identification in the Literature

Main Methods |  References & Year |  Description
---|---|---
Face Morphing |  2013 (Korshunov and Ebrahimi, 2013b), 2015 (Nakashima et al., 2015), 2017 (Mahajan et al., 2017), 2019 (Li and Lyu, 2019) |  Synthesizing a face that maintains a likeness to an input and a target face using various techniques.
Face Synthesis |  2005 (Newton et al., 2005), 2018 (Sun et al., 2018), 2019 (Karras et al., 2019; Li and Lin, 2019), 2020 (Le et al., 2020) |  Synthesizing realistic faces that preserve generic facial characteristics using $k$-same algorithms that compute an averaged face representing a cluster of $k$ similar faces or using DL-based synthesis.

Face Morphing. Face morphing (illustrated in Fig. 9) is a geometrical
transformation involving two face images to create a manipulated image that
maintains a likeness to each of the original identities. In (Korshunov and
Ebrahimi, 2013b), Korshunov et al. (2013) proposed a partial and randomized
face morphing method that works by interpolating the position and intensity of
selected key pixels in the input face to match a target face (Rakhmawati et
al., 2018; Korshunov and Ebrahimi, 2013b). The interpolation of the key pixels
is achieved somewhere in-between the input and target images, and the pixel
intensities are weighted. First, a face landmark localization algorithm
automatically selects pixels around the detected eyes, nose, and mouth from
both images. The original and target images are divided into triangles using
Delaunay triangulation, with the selected points being the vertices of the
triangles. For corresponding pixels in both images, the final pixel coordinate
is determined using a given level of interpolation value. For an interpolation
level and intensity strength equal to 0, the original pixels are generated,
and for a level of 1, the target image is produced. Recovery of the original
image is possible by applying the inverse morphing transformation, given
knowledge of the target image, selected key points, interpolation values, and
intensity values. Security can be ensured by using a secret key for seeding
the pseudo-randomization of the interpolation and intensity weights and
encrypting the key points used for the triangulation.

The previous method lacked naturalness due to partial morphing. To achieve a
higher utility, Nakashima et al. (2015) in (Nakashima et al., 2015) proposed a
method to preserve facial expression based on image-melding. Compared to the
previous method, this technique leverages only a smaller number of
corresponding points, specified manually by the user, in target and source
images of similar orientation. The corresponding points in the source image
are roughly matched to the target’s by a transformation process, applying a
weighted mask. The mask adjusts the scale, orientation, and position of the
source image’s facial features. Mahajan et al. (2017) (Mahajan et al., 2017)
developed a system for face swapping. The system works by directly fitting a
secondary image over the input image by rotating and scaling it. Color balance
adjustment and blending of the facial landmarks from the second image onto the
input are made to achieve a more natural appearance. In (Li and Lyu, 2019), Li
et al. (2019) created a Facial Attribute Transfer Model (FATM) using
autoencoders. The DNNs blend the target’s facial attributes to those of the
original faces.

Figure 9. Face Morphing (Seibold et al., 2017): (left, right: original faces,
center: morphed face).

Three faces where the last face is a combination of the first two on the left.

Face Synthesis: Here, we discuss $k$-same and GAN-based methods. Both methods
rely on different mechanisms to effectively produce a new face image, unlike
face morphing that modifies an input face to match a target face.

$k$-same Family. The basis of $k$-same algorithms is $k$-anonymity, an
extension from differential privacy for micro-aggregated data. Other data
anonymization techniques exist for categorical data; the two most popular
among those are $l$-diversity and $t$-closeness (Le et al., 2020). $k$-same
algorithms compute an averaged face to represent a cluster of $k$ similar
faces. The first work to introduce this concept was by Newton et al. (2005)
(Newton et al., 2005) to enable secure sharing of image data with provable
guarantees. The algorithm first determines the similarity between faces and
clusters them based on a distance metric. The authors proposed pixel-wise
averaging ($k$-Same-Pixel) or eigenvectors averaging ($k$-Same-Eigen) (Newton
et al., 2005). Images produced by $k$-Same-Eigen have a blurred effect because
only a minimal number of important facial characteristics is retained. Even
though the averaged face may appear similar to one image than another, because
the same averaged face replaces all images in the cluster, the method
guarantees that the correlation success rate of the $k$ faces is $\frac{1}{k}$
at most. The main challenges are determining the appropriate value for $k$ to
achieve the highest privacy and finding the optimal groups of closest faces,
which is an NP-hard problem (Newton et al., 2005). The main limitation of
(Newton et al., 2005)’s method is its lack of naturalness and information loss
during the de-identification process.

Many additions to the $k$-same family have been proposed to overcome some of
the previously mentioned limitations. Gross et al. (2005) (Gross et al., 2005)
presented $k$-Same-Select, which guarantees the utility of the data by
preserving facial expressions or gender. However, $k$-Same-Select works in the
pixel domain and sometimes suffers from alignment mismatch in the de-
identified face. $k$-Same-M by the same authors (Gross et al., 2006) addressed
this limitation by using statistical models instead. Meng et al. (2014) (Meng
and Sun, 2014) proposed $k$-Same-furthest, which aims to achieve perfect
privacy for any source face regardless of the value of $k$. Sun et al. (2015)
proposed $k$-Diff-furthest (Sun et al., 2015) to solve the tracking problem of
individuals in $k$-Same de-identified videos, where $k$ similar faces are
replaced with the same averaged face. More recently, two studies incorporated
the concept of $k$-Same algorithms in neural networks and GAN-based face
synthesis in (Meden et al., 2018) and (Pan et al., 2019), respectively.

GAN-based Methods. Images produced by the early solutions of $k$-Same may
exhibit poor visual quality and ghosting artifacts due to inexact alignment of
facial features (Le et al., 2020; Croft et al., 2019; Bursic et al., 2021).
Generative Adversarial Networks (GANs) are more successful in producing
natural-looking face images (Kim and Yang, 2020) that preserve many facial
characteristics and cannot be reliably linked to the original identity.
However, face detection is problematic when the subjects appear in varied
backgrounds, positions, and activities. Most generative face models are suited
for frontal and strictly aligned faces. To tackle this problem, Sun et al.
(2018) (Sun et al., 2018) devised a two-stages solution. Depending on the
visibility of the face pixels, facial landmarks are either detected or
generated. They are detected when the original face image is clear, but
hypothetical realistic facial structures and poses are generated when the face
has been obfuscated. An advantage of the second scenario is that it allows
upgrading weak obfuscations, including blocking and blurring, to use realistic
synthetic faces that blend naturally into the context.

Le et al. (2020) (Le et al., 2020) designed a system based on StyleGAN (style-
based GAN) (Karras et al., 2019) and $k$-same algorithms to achieve provable
privacy, which synthesizes a face for a given dataset while simultaneously and
iteratively tuning its privacy-utility trade-off. The system uses a two-steps
cluster analysis component for feature vector extraction and clustering all
input images into fixed-sized sets. The authors report that low-dimensional
feature vectors produce better clustering and ensure high naturalness. The
synthesized faces preserve properties such as age, gender, skin tone, and
emotional expressions. Finally, the authors used a measure of the average
Euclidean distances between images in a cluster and the corresponding
synthesized image (i.e., measuring total information loss). StyleGAN (Karras
et al., 2019) architecture is superior to traditional GAN in terms of the
established quality metrics and the unsupervised separated learning of high-
level attributes (e.g., pose, freckles, hair). Another noteworthy work is
AnonymousNet (2019) (Li and Lin, 2019) which uses GANs to produce realistic
synthetic faces but notably adds adversarial perturbations to reduce the
recognition accuracy and provides provable privacy metrics through a
$k$-anonymity-based attribute selection.

####  2.1.4. Adversarial Perturbation

Studies have shown that advanced ML systems can, in partially obfuscated
images, make correct malicious inferences about the person, such as religion,
age, and occupation, thus deeming visual obfuscation methods insufficient when
adopted in isolation (Oh et al., 2016). Adversarial perturbation mechanisms
were developed to change the prediction of deep neural networks, so the
adversary draws incorrect or specific intended inferences from the image data
without making perceptible changes to the input image. Another advantage of
this technique is that it can be applied to an already obfuscated image to
strengthen the obfuscation further. Recent works have developed the concept of
Universal Adversarial Perturbations, which can be added to any image and cause
the target ML model to misclassify it. Universal perturbations are more
practical to deploy than per-instance perturbations since minimal computation
and knowledge (black-box attack) are needed (Chaubey et al., 2020).

Some approaches to crafting adversarial face images lack quality and take an
unreasonable amount of time to be generated. Deb et al. (2019) (Deb et al.,
2019) proposed, AdvFaces, an automated adversarial face synthesis method that
learns to generate minimal perturbations in the facial regions via GANs.
AdvFaces produces imperceptible perturbations that evade recognition with high
success rates up to 97.22% for obfuscation attacks (falsely rejecting a
genuine subject) and 24.30% for impersonation attacks (falsely accepting an
imposter) (Seibold et al., 2017; Scherhag et al., 2019). To contrast image
classification tasks, Li et al. (2019) (Li et al., 2019) proposed an algorithm
that generates adversarial noise using a novel adaptation of Fast Gradient
Sign Method (FGSM) (Sanchez-Matilla et al., 2020), which limits the
probability of inferring the true class of a distorted image successfully.
Different variations of FGSM such as iterative FGSM, private FGSM, random
FGSM, and least-likely FGSM may select the true class as the target class. The
authors designed the new FGSM approach to pick the target class from an
adaptive subset of classes that most likely does not include the class to be
protected. A recent survey by Chaubey et al. (2020) in (Chaubey et al., 2020)
provides a comprehensive summary of the existing adversarial models and
defense mechanisms against them. Other examples of works based on image
adversarial methods are referred in (Hukkelås et al., 2019; Raval et al.,
2017; Oh et al., 2017; Rajabi et al., 2021).

####  2.1.5. Abstraction

Abstraction removes an object of interest and replaces it with a graphical
representation that somewhat resembles the removed content. Fig. 10 shows
three examples of object abstraction found in the literature: silhouette,
point-light (or skeleton), and avatar. Since abstraction is a utility-and-
privacy-preserving method, it is widely used in online social networks. An
abstraction algorithm applies image inpainting methods to remove the segmented
object of interest, renders a visually similar object, and places the
abstracted form over the exact pixel locations of the removed ROI.

The effectiveness of the following abstraction methods was investigated by Li
et al. (2017) in (Li et al., 2017). The point-light or skeleton method
replaces an ROI with points and lines corresponding to the ROI’s shape
outline. A human point-light can only illustrate a person’s activity and
height, but completely hides his identity. The silhouette method is where an
ROI is replaced with a (monochrome) object that mirrors its shape. A human
silhouette can preserve the subject’s body pose, activity, clothing style,
gender, health condition, or hair style, but still completely removes
biometric identifiers. A human avatar can preserve, in addition to all
previously listed features, finer details about the person such as facial
expressions and age, and still protect the identity from disclosure. The
privacy-utility balance is vividly exhibited in the previous examples. Unlike
face replacement methods, abstraction does not aim to preserve naturalness;
however, it seeks to generate realistic representations of the removed
sensitive region. Nonetheless, abstraction is still considered one of the most
effective methods for visual obfuscation today.

Figure 10. Abstraction techniques (Padilla-López et al., 2015): (left: solid
silhouette, center: point-light, right: avatar).

A full body photo of a human in an office; where the body was abstracted out
in three different methods.

####  2.1.6. Blocking / Masking

Blocking or blacking (see Fig. 11-b) is where the sensitive bounding box is
detected and replaced with a shape (e.g., square, rectangle, ellipses) in a
solid color, typically black. Whereas the term masking is similar to blocking
and is frequently used interchangeably in the literature. However, we find
that it commonly refers to less intrusive covering of the sensitive regions,
e.g., the bar mask in Fig. 11-c, where the eyes are determined as the
sensitive area of interest. It may also refer to covering the sensitive
regions with other objects, e.g., covering a face area with a clip-art (Yuan
et al., 2015a). Noise addition can be considered a subtype of image masking
(Newton et al., 2005). In noise addition, random image pixels are replaced by
black and/or white pixels (referred to as salt-and-pepper noise (Seibold et
al., 2017)) as seen in Fig. 11-e. Black/white images are modified by flipping
random pixels, and in gray-scale images (see Fig. 11-d), a random value
between 0 and 255 replaces random pixels. However, additive random noise does
not thwart automated recognition except when roughly 50% of the pixel values
have been flipped in black/white images or 75% of the pixels are changed in
gray-scale images (Newton et al., 2005). Moreover, the black bar mask around
the eyes is ineffective against image inpainting attacks, e.g. (Barnes et al.,
2009; Yeh et al., 2017; Zhao et al., 2019). Blocking and masking are the most
utility-reducing among all other ad-hoc obfuscation methods (Vishwamitra et
al., 2017). Nonetheless, blocking offers high protection against human and
automated object recognition (Tekli et al., 2019).

(a) Original

(b) Block

(c) Mask

(d) Grayscale
Noise

(e) Black- white noise

Figure 11. Examples of Blocking/Masking (Newton et al., 2005)

Ad-hoc masking or blacking of images to obscure the faces in them.

####  2.1.7. Inpainting

Inpainting methods are generally used to repair damaged or missing parts of
photos. However, inpainting as an obfuscation method (see example in Fig. 12)
refers to completely removing sensitive content (e.g., full-body region) of an
image. The cleared space is then filled to blend naturally in a visually
consistent manner, appearing as part of the background (Li et al., 2017).
Based on the study conducted in (Li et al., 2017), inpainting is the most
effective technique for achieving true privacy. In the cited study, it
produced the smallest percentage of identification success rate amongst 13
other obfuscation methods. The output of inpainting algorithms is becoming
more seamless and photo-realistic with the use of deep learning and
adversarial training approaches. Once the image is inpainted, a visual
abstraction of the removed sensitive information (e.g., (Padilla-López et al.,
2015; Blažević et al., 2015)) or synthetic faces (e.g., (Sun et al., 2018))
can be added to prevent complete information loss. A major limitation of
inpainting algorithms is their computational complexity (Rakhmawati et al.,
2018).

Figure 12. Image Inpainting (Padilla-López et al., 2015): original image
(left) and inpainted image (right).

On the left side, a woman is posing for a picture in front of a waterfall. On
the right side, the same picture is shown except the woman has been removed,
leaving only the waterfall background.

###  2.2. Attacks and Countermeasures

Image obfuscation alters or removes features from the images to hide sensitive
information and retains some visual features to keep the image intelligible
and suitable for processing. However, these visual features can still be used
to identify or reconstruct the protected objects through different attack
styles. The attacks against image obfuscation can be classified as human-
based, machine-based, recognition-based, reconstruction-based, or inference-
based. In this section, we focus separately on recognition (or re-
identification) attacks, reconstruction (or restoration) attacks, and
inference attacks.

Recognition-based Attacks. Recognition-based attacks are where an attacker
attempts to re-identify obfuscated objects or the person(s) from their
obfuscated faces. Human-based recognition is based on the degree of
information loss in the obfuscated image and the attacker’s familiarity with
the scenes and objects before the obfuscation. Removal methods such as
blocking, masking, and inpainting may completely prevent human recognition (Li
et al., 2017; Vishwamitra et al., 2017). Conversely, image distortion methods
(e.g., image filters, warping, false colors, cartooning) preserve the general
shape and intensities of the objects, and therefore, are not very resilient to
human detectors (Li et al., 2017). There are several ways to qualitatively
measure an identity disclosure risk. A definitive privacy leak is when an
image can be correctly linked to an individual with high certainty. Potential
privacy risks exist when private information is linked to several individuals.
The highest level of privacy is when an image cannot be related to any
individual.

In terms of machine-based recognition, an attacker would use machine learning
algorithms for image recognition. Three different kinds of automated attacks
involve the matching of input and output images in various arrangements: (a)
original to obfuscated; (b) obfuscated to original; and, (c) obfuscated to
obfuscated (Newton et al., 2005). In the first type of attack, the obfuscated
images are run directly through the image recognition software. In this
context, the recognition algorithm uses a training set (aka, gallery images)
composed of the original images, while the obfuscated images (aka, probe
images) are used as input. The second type is the dual of the first; the
training set comprises obfuscated images, and the original images are used as
input. This attack assumes that the attacker has the complete collection of
original images. According to (Newton et al., 2005), the trained model is
expected to have decomposed and dispersed the alterations on the obfuscated
images. The work in (Guo et al., 2017) serves as an example of this attack,
where input transformations on perturbed images counteract the effect of
adversarial perturbations. In the third attack, the attacker invokes the same
obfuscation technique on a set of probe images. The recognition is done by
matching an obfuscated probe to an obfuscated gallery. This attack assumes
that the adversary has access to the training set obfuscated by a given
method. This is also called parrot attack or imitation attack; a process of
comparison and elimination to recognize the obfuscated image. In (Padilla-
López et al., 2015), pixelization and blurring were found to be vulnerable to
parrot recognition which can achieve recognition rates close to 100%, despite
looking somewhat de-identified to human recognizers (Padilla-López et al.,
2015). It is shown that parrot attacks drastically reduce the level of privacy
protection and improve the recognition rate (Ribaric et al., 2016; Ribaric and
Pavesic, 2015).

To counter image recognition attacks, adversarial perturbations can be added
to alter the machine-learning based prediction, while having the introduced
perturbation not detectable by the human eye. However, there are some defense
mechanisms suggested in the literature for detecting and rectifying such
perturbations. Two of such defenses are considered: the first one is to pre-
process images at the inference time (Guo et al., 2017), and the second one
relies on increasing the robustness of the model against adversarial training
(i.e., training with perturbed images) (Chaubey et al., 2020). In (Akhtar et
al., 2018), Akhtar et al. (2018) presented a rectification method called
Perturbation Rectifying Network (PRN) that pre-processes the images to be
classified before passing them to the target recognition model. The output of
PRN is a rectified image minus the perturbation. A detector network,
separately trained on the DCT of the input-output difference of the PRN,
performs a binary classification to decide if an adversarial perturbation is
present. If perturbed, a rectified image is passed to the target model instead
of the probe image. PRN is shown to have a success rate up to 97.5% against
adversarial perturbations. Another possible countermeasure to recognition
attacks is pseudo-randomization (based on a secret key) in the de-
identification algorithm. The randomization will impede an attacker’s ability
to replicate the de-identification process exactly. When attaining high
privacy is the main concern, we recommend not to use standard image
transformations (such as pixelization, blurring, masking, lossy compression,
and other transforms), even though they have lower computational complexity,
since they are ineffective against ML attacks.

Reconstruction-based Attacks. Reconstruction-based attacks aim to recover the
original image before its transformation. Today, with the advent of deep
neural networks, this task has become much less challenging. Several studies
have shown that conventional image transformations, namely pixelization,
blurring, masking, and compression, are ineffective against bicubic
interpolation, statistical modeling, and DL-assisted attacks (McPherson et
al., 2016; Tekli et al., 2019; Hill et al., 2016).

For pixelized and compressed artifacts, a study in (Ledig et al., 2017) used
neural networks to achieve image super-resolution (Wang et al., 2020; Anwar et
al., 2020), upscaling images up to a factor of 4. In further works, GAN-based
models were used to solve super-resolution problems (Wang et al., 2018), and
the study in (Yu and Porikli, 2016) achieved ultra-resolution for images of
lower resolution (e.g., 16$\times$16). Similarly, for canceling the impact of
blurring and restoration of sharper images, several image-deblurring methods
were proposed in the literature using neural networks (Koh et al., 2021), and
GANs (Kupyn et al., 2018).

Image inpainting can be used to infer missing pixels for blacked-out sections
in an image. Yeh et al. (2017) (Yeh et al., 2017) trained a deep generative
(GAN-based) model to find an encoding in the latent space that is closest to
the corrupted image. The generator then uses the encoding to reconstruct the
missing parts. The major advantage of their method is that it works with
arbitrarily structured and large masked regions. Another method in the
literature was also successful in recovering blacked image regions, however,
relying on the presence of similar patches and structures in the remaining
parts of the input image (Barnes et al., 2009). Other solutions used
statistical modeling such as total variation and low rank and machine learning
such as nearest neighbor (NN) and autoencoders (Yeh et al., 2017). In
comparison, semantic image inpainting produced much more realistic images with
sharper edges, having higher PSNR and SSIM in randomized masks, even when 80%
of the pixels were missing. Prediction of the missing pixels can be done using
external data (e.g., guidance images (Zhao et al., 2019)) or internet-based
inpainting (Whyte et al., 2009).

Inference Attacks. As the name suggests, an adversary can infer hidden
attributes or information about the obfuscated image in an inference attack.
Inference includes determining that an image itself has been manipulated or
synthesized. This is specifically valid for naturalness-preserving obfuscation
technologies, such as face morphing and face synthesis. In face de-
identification, indistinguishability is an important property where the
synthetic face should look realistic and not be easily differentiated from a
natural one. Examples of works dealing with image forgery exist; detection
through analysis of reflections and shadows or binary statistical features,
detection of adversarial images (Hendrycks and Gimpel, 2016), or detection of
deep fake images (Tolosana et al., 2020).

###  2.3. Discussion

Applications. The need for image obfuscation is urgent and evident, given the
amount of images that are circulated and shared daily on the Internet or over
Online Social Networks (OSNs). Due to the privacy-intrusive nature of the
Internet, the community was driven to advance the research in privacy-
preserving and privacy-aware systems and their applications. Examples of such
systems and applications are: (a) image processing, search, and retrieval on
the cloud; (b) image capture in video camera surveillance, health monitoring
applications, IoT sensor networks, Internet of Multimedia Things (IoMT), and
commercial applications such as Google Street View; (c) image sharing and
publication on OSNs; and, (d) image classification and recognition for
automated tasks, such as autonomous driving and supply chain optimization
(Tekli et al., 2019).

Challenges and Open Research. Undesirable visual exposure and malicious
inference of high-level sensitive information are standard image privacy
problems (Liu et al., 2020). According to previous studies, implicit
information such as health conditions, social class, and relationship status
can be deduced from images using ML models, leading to privacy infringement.
However, protecting against malicious inference remains challenging due to the
multi-modality of the underlying features, which are typically tricky to
recognize using conventional image recognition. The field of multimodal
attribute detection and obfuscation in images is still in its infancy; only a
few studies addressed the perturbation of hairstyle, clothing, race, and
gender in images. In image de-identification, face and body detection
significantly impact the achieved protection; that is, failure to locate
several soft-biometric identifiers results in a high re-identification risk.
Face detection challenges include partial visibility of the face regions due
to occlusions, accessories, sunglasses, and variations in head pose, lighting,
and backgrounds, which can obstruct the detection algorithm’s ability to
recognize face regions correctly. Current studies still try to mitigate those
recognition challenges. In addition, other studies aim to generate realistic
faces that maintain a high degree of similarity in terms of facial expressions
and head pose and mimic body poses to relay an accurate representation of the
human activity being performed (Le et al., 2020). There is also a need for
quantifying privacy and utility through objective evaluation.

Key Properties. The key properties to consider when selecting an optimal
obfuscation method for protection against information leakage are (Li et al.,
2019): (a) naturalness and utility: the distortion should be unnoticeable; (b)
(ir)reversibility: (in)ability to undo the transformation; (c) robustness and
security: thwarting most human and machine-based attacks; (d)
adjustable/adaptable: protection with different degrees of strength; (e)
provably secure: provides measurable security metrics based on differential
privacy or distance metrics; and, (f) computational complexity: it should be
easy to implement in terms of resource usage. The study field of image
obfuscation has reached a mature level, where the latest techniques are based
on adversarial learning. Therefore, rather than developing new obfuscation
methods, most research is now focused on improving the utility of the
adversarial training output while keeping security/privacy a priority. When
one refers to the “utility” of an obfuscated image, they are typically
concerned with three things: its naturalness (i.e., Does it look realistic?),
its intelligibility (i.e., Does the distortion impact the ability to infer
important attributes?), and other context-specific properties (e.g., generated
obfuscations for images with faces belonging to different people should be
distinguishable). The first three properties introduced at the beginning of
this paragraph (naturalness, reversibility, and security) are addressed in
most image obfuscation methods. However, the remaining ones are selectively
implemented based on the user requirement. In terms of provable privacy
guarantees, the main way to achieve it is by applying statistical and
aggregated data protection methods derived from differential privacy such as
$k$-anonymity. However, most existing protection solutions do not provide
measurable privacy guarantees. Moreover, selective obfuscation (i.e., applying
the obfuscation on a per-feature basis), adjustable privacy filters, and
automatic evaluation of image privacy and utility, are still ongoing research
topics.

Review of Methods. In Table 3, we have given low, medium, and high ratings for
the main categories of image obfuscation in terms of their utility, the
ability to reverse the obfuscation and protection level against human and
machine attackers. We further explain the rating choice here. Simple ad-hoc
image transformation methods (blurring, pixelization, lossy compression, edge,
warping, style transfer, false color, cartooning) distort sensitive ROIs in a
way that results in a high degree of information loss. Still, at the same
time, their output maintains a strong resemblance to the original general
shapes of the image. However, aesthetic transforms (false colors, style
transfer, and cartooning) provide a better visual experience than the others.
These methods are no longer viable due to robust ML recognition (Newton et
al., 2005) that can deduce the original input before its transformation.

On the one hand, encryption and scrambling completely hide the ROI and,
therefore, adversely impact the image’s utility. On the other hand, they allow
full recoverability of the original content using a secret key. The security
of the encryption-based protection depends on the key strength and the
permutations’ strength. Possible solutions to break the protection are based
on brute force and are computationally complex.

Suppression methods such as masking, blocking, and inpainting altogether
remove the sensitive ROIs, resulting in the highest degree of information
loss. Although, inpainting is less visually intrusive and produces a more
natural effect. These methods are robust against human and machine
recognition. Studies have shown that image inpainting methods can be used to
fill in missing (masked) areas.

Abstraction entirely replaces an ROI, thus protecting its identity with a
visually similar representation. The utility of the replacement depends on the
replacing object, its quality, and its resemblance to the removed ROI.
Overall, inpainting and abstraction may be most useful as obfuscation methods
because they are both effective at increasing privacy for elements of an image
and providing a good viewer experience (Li et al., 2017). This shows that to
achieve a higher privacy-utility balance, more processing time and
computations are needed. Machine learning can provide efficient obfuscation
models; however, their accuracy depends on how extensive the training set is.

In terms of advanced obfuscation, the solutions are mainly based on face
synthesis and adversarial perturbation. Adversarial perturbations are not
perceivable by humans; they add little visual distortion to the image. Their
purpose is to deceive automated machine recognition. However, some studies in
the literature have shown that adversarial perturbations are detectable and
can be countered using ML methods. Though, the cited countermeasures assume an
attacker with access to the perturbed training set, which is not always
possible in real-world scenarios. Face synthesis is a non-reversible method of
generating new faces by averaging a cluster of similar faces ($k$-same family
algorithms) or adversarial learning. The method effectively reduces the re-
identification risk and typically generates natural-looking faces that
preserve gender, facial expressions, age, or race while protecting the
identity.

##  3\. Video Obfuscation

Video surveillance systems are today’s most ubiquitous source for video
capturing people and their activities (Sah et al., 2017; Ribaric and Pavesic,
2015). As expected, the public use of video surveillance (including CCTV and
drone surveillance) introduces the need for privacy protection, specifically
for law-abiding citizens, while providing the authorities the ability to
retrieve the original video for specific purposes, e.g., crime investigation.
When releasing the video, portions of the footage exposing sensitive
information are redacted. Such information include but is not limited to
faces, license plates, tattoos, clothing, computer screens, and house numbers
(Sah et al., 2017). Videos are a series of time-related image frames; hence,
the expectation is that image de-identification techniques are applicable to
videos. However, the process is inherently more complex, since neighboring
image frames in a video sequence are closely related.

Object detection and tracking accuracy play a significant role in identifying
and obfuscating sensitive regions correctly. Any incorrect detection of
sensitive areas in a single frame can destroy the protection for the entire
video sequence (Sah et al., 2017; Korshunov et al., 2012). Object detection
and tracking in video sequences also face further challenges, such as:
variation in the object’s localization and size throughout the scenes; bad
lighting conditions; variable illumination; cluttered scenes; and, partial
occlusions due to obstructing components (e.g., sunglasses, accessories, and
facial hair on face regions) (Ribaric et al., 2016; Ribaric and Pavesic,
2015). Additionally, some video surveillance systems require real-time
detection and de-identification of moving objects. For that reason, the
selection of the obfuscation method is critical since real-time processing
should be fast and effective. In general, for most video surveillance systems,
the requirements to be satisfied are: (a) real-time processing; (b) efficiency
and low cost; (c) utility-preserving obfuscation (preserving gait, physical
actions, and facial gestures); (d) reversible obfuscation; (e) adaptive
obfuscation (adjustable levels of privacy filters parameters, or selective
obfuscation based on preset rules); and, (f) format compliance (decodable by
commercial video players).

###  3.1. Techniques

Various approaches have been proposed for privacy protection in videos. The
most common solution is applying visual transformations on sensitive image
regions. Other approaches, which are out of scope but are defined briefly
hereafter, include physical intervention, software intervention, and secure
multiparty computation (Padilla-López et al., 2015). Physical intervention
creates capture-resistant regions using specialized hardware (e.g., adhesive
camera blockers, pulsating LEDs directed at camera lenses). In contrast,
software intervention is achieved through modifying the firmware of a
capturing device to prevent capturing specific scenes (Padilla-López et al.,
2015). Secure multiparty computation allows image processing tasks (e.g., face
detection, object tracking, image segmentation) to be computed anonymously.
The traditional approach for visually transforming sensitive regions in videos
is manually identifying, annotating, and masking ROIs in each video frame.
However, considering a typical surveillance video of 30 frames per second,
manual redaction is unpractical being extremely time- and labor-intensive
(Ribaric and Pavesic, 2015). Automatic obfuscation can target specific objects
or the entire image frame (aka, general/global obfuscation). The latter
assures higher protection yet it removes the contextual information necessary
for monitoring people and their behavior. Even though our main focus in this
subsection is object obfuscation, we still give a brief overview of the state-
of-the-art object detection and object tracking, focusing on face objects,
since they are closely related to the obfuscation process. Next, we review
image obfuscation methods in terms of how suitable they are for real-time
surveillance.

####  3.1.1. Object Detection

This step is needed for the automatic localization of relevant objects in a
frame. The detection algorithm identifies the object’s bounding box, and
segmentation obtains the precise pixels outlining the object within the
bounding box. There are four main approaches to object detection: sliding
search window, region proposals, DL-based, and pixels-based (Sah et al.,
2017). The sliding window approach sweeps a detection template across an
image, computes a function with respect to the template, and classifies the
image region. It is a computationally demanding method and, therefore, it is
not typically used for video redaction. Region proposal methods use
classifiers (e.g., SVM, k-nearest neighbor) that obtain a confidence score for
each candidate image region using its extracted low-level features (e.g.,
SIFT, histogram of oriented gradient (HOG), Harris Corners, etc.). The
frequently used deep learning models for object detection include Regional CNN
(RCNN), Fast RCNN, Mask RCNN, Multibox, and YOLO (You Only Look Once) (Sah et
al., 2017; Asghar et al., 2019). YOLO uses regression models to predict the
coordinates of the bounding box and its class confidence by applying a single-
pass CNN (Sah et al., 2017). Pixel-based methods achieve semantic segmentation
and assign a class label to each pixel in a frame. Some approaches use fully
convolutional neural networks (FCN) and Conditional Random Field (CRF) to
achieve per-pixel classification (Sah et al., 2017; Qin et al., 2017).

In terms of face objects, face detection algorithms (Zafeiriou et al., 2015)
are based on feature-level or image-level methods. The feature-based approach
uses low-level analysis (edges, skin color, motion, gray-scale), active shape
models (snakes, deformable templates, PDM), or facial feature searching and
extraction (constellations, Viola-Jones, Gabor Feature). The image-based
approach is a learning process where a trained model classifies images as face
or non-face, e.g., using linear subspace methods (Eigenfaces), statistical
methods (GMM, PCA, SVM), and neural networks (Ribaric and Pavesic, 2015; Chen
et al., 2009). A recent survey providing a comprehensive overview of face
detection techniques is referred in (Kumar et al., 2019).

The majority of automated face detection algorithms are limited by head poses,
occlusions, image resolution, and so on. To mitigate the inaccuracy of some
face detection software in suboptimal conditions, Chen et al. (2018) (Chen et
al., 2018) proposed ReSPonSe—a system that uses human trace tracking instead
of face detection to identify the pixels representing facial information in a
more robust way. Human trace tracking gives higher assurance that the face and
the body regions are still identifiable by color-based skin detection when
faces are not detected. The system architecture has two main stages, the
encapsulation stage, and the decapsulation stage. In encapsulation, the video
sequence is processed in real-time to remove private information using human
trace tracking and image encryption. Human trace tracking consists of: face
block extraction to accelerate face detection; in-block face detection to
detect faces in the extracted blocks; and, detection rectification. This step
also utilizes the distribution of skin regions in image tiles surrounding
selected key blocks. The extracted detections are then encrypted. In
decapsulation, decryption is performed on the redacted video to selectively
recover the original faces based on the viewer’s certificate. Human trace
tracking does not work well when non-face objects (e.g., walls or cars)
possess a color similar to skin colors; it results in the unnecessary
redaction of the background, thus reducing the utility of the video.
Therefore, Chen et al. (2019) (Chen and Nahrstedt, 2019) proposed the Facial
Information Segmentation (FIS) algorithm, which combines color information,
Harris Corner, and face detection algorithms to identify the pixels associated
with the face region. The authors rely on the observation that the density of
Harris Corners is lower on non-face objects than other face objects with skin
color; non-face objects tend to be smoother and have fewer features.

####  3.1.2. Object Tracking

The purpose of object tracking is to estimate and predict the size and
location of a detected (and tagged) object over time in subsequent video
frames. Tracking algorithms are susceptible to noise factors due to variations
in illumination, object scale, occlusions, pose or camera relative
perspective, and motion blur. There are four main approaches to object
tracking: motion-based; appearance-based; tracking by detection; and, deep
learning methods (Sah et al., 2017). Motion-based modeling computes an
estimation of the stationary scene background and subtracts the estimated
background from each frame having a moving object. The differences calculated
from the background subtraction between consecutive scenes indicate the
presence of objects in motion. Appearance-based matching of hand-crafted or
machine-learned image features includes color histogram matching, shape
matching, or texture matching. Again, it is limited by dramatic changes in
camera perspective, heavy shadows, etc.. Tracking by detection extends the use
of object detection algorithms to enable tracking frame by frame (e.g., using
adaptive search windows). However, this method has a high computational
overhead and may not be suitable for real-time processing. Lastly, more
advanced techniques such as recurrent neural networks exploit the history of
an object’s location to learn temporal dependencies between frames.

In the case of face tracking, to preserve naturalness for de-identified
videos, deep learning methods for face landmark localization (locating facial
key points around the eyes, corners of the mouth, and nose) and motion and
pose estimation are considered (Ribaric et al., 2016; Asghar et al., 2019).
Additionally, the algorithm should discriminate and predict the location of
all moving faces in the video sequence. Although using temporal information
for tracking increases robustness, it is ill-suited for real-time tracking. In
some solutions, markers or RFID tags worn by the individuals were used to
enhance tracking accuracy (Sah et al., 2017; Cheung et al., 2009).

####  3.1.3. Obfuscation Methods

Video surveillance systems follow the same protection techniques used in still
images. Masking, pixelization, and blurring are standard methods for video
redaction since they do not require an excessive processing power and can be
done in real-time. Korshunov et al. (2012) (Korshunov et al., 2012) performed
a subjective evaluation to assess the detection accuracy of abnormal behavior
in video surveillance scenes when the previous methods were applied to body
silhouettes and faces. The results showed that the recognition of the
obfuscated people was still possible to a certain degree, specifically the
gender attribute from the shape of the body image in the video, even when the
masking filter is applied. Pixelization was found to offer higher utility in
terms of action recognition than blurring in this context, and masking overall
provides the highest privacy protection because it can reduce recognition
accuracy to near-zero (Korshunov et al., 2012; Korshunov and Ebrahimi, 2014).
In (Ivasic-Kos et al., 2014), Ivasic et al. (2014) studied the impact of
Gaussian Blur filters on human activity recognition and de-identification. The
results show that specific activities (e.g., jump in place and wave one hand)
reveal more discriminant information that can be used for subject
identification than others (e.g., walking and running). The authors suggested
increasing the variance of the Gaussian blur for such activities.

There are other methods that are also not as demanding for processing power,
while being more utility-preserving. For instance, cartooning (Erdélyi et al.,
2014), edge detection (Chen et al., 2009), style transfer (Brkić et al., 2016,
2016), false colors (Çiftçi et al., 2015), warping (Korshunov and Ebrahimi,
2013b), and morphing (Korshunov and Ebrahimi, 2013a). In the following, we
introduce some examples from the literature. In (Chen et al., 2009), Chen et
al. (2008) applied edge motion history image (EMHI) to obscure body images
while still capturing the structure of the bodies and their motion using
edges. Korshunov et al. (2014) (Korshunov and Ebrahimi, 2014) evaluated
morphing and warping against blurring, pixelization, and masking. Warping is
the least suitable filter because it affects recognition only at very high
distortion levels, whereas it is also more complex than the other simple
filters. The authors concluded that the morphing filter seems to be the best
choice among the evaluated privacy filters for video surveillance since it is
reversible, and recognition accuracy can be varied linearly by varying the
pixels’ intensity weight. False-colorization is a general obfuscation method
applicable to video surveillance since it is reversible. It may also hide
identity information without impacting surveillance-related information, such
as the count of people in an area.

Lightweight partial encryption (i.e., ROI-based) and scrambling (Dufaux and
Ebrahimi, 2008; Ruchaud and Dugelay, 2016) are other applicable methods to
real-time processing, even though they are less secure than naïve encryption
(Padilla-López et al., 2015). The videos are obfuscated in a way that they
remain format-compliant and viewable in the distorted form. Other advanced
methods such as object replacement (e.g., avatars (Blažević et al., 2015)) or
face replacement can be used; however, both require precise position and pose
tracking (Cheung et al., 2009). Blavezvic et al. (2015) in (Blažević et al.,
2015) proposed a reversible real-time de-identification pipeline that
obfuscates all biometric identifiers in video footage. The system relies on
computer vision algorithms, namely human body image detection, segmentation,
and tracking of the detections. The detected human body in the original image
is replaced with a 3D avatar. The 3D model is rendered precisely on top of the
original body with the help of human pose estimation (joint modeling)
algorithms. The detection and pose estimation strongly depends on the viewing
angle, body shape, clothing, etc.. The de-identified region is encoded and
embedded in the carrier image using steganography to enable reversing the
concealment. The authors identified some limitations in their 3D modeling;
specific body movements are not supported, such as jumping or rotating. In
addition, sudden movements are not always accurately tracked. The likelihood
of erroneous output and incorrect localization of some joints (and therefore
incorrect positioning of the 3D avatar) due to the tracking system is
relatively high. To countermeasure the tracking inaccuracies, the authors
pixelized and blurred the body sections visible behind the avatar in the
original frame. Further works included developing more natural-looking
avatars.

###  3.2. Attacks and Countermeasures

Since the same protection methods used for images are used for videos, the
same attacks described in Section 2-B apply. In the following, we explain the
major differences in the recognition-based and reconstruction-based attack
styles for videos.

Recognition-based Attacks. Recognition-based attacks (or re-identification
attacks) are also a significant concern about de-identified videos. Full-body
images are more commonly seen in surveillance videos, and even when face
masking is used, a person’s identity can be exposed using pairwise constraints
attack (Sah et al., 2017; Chen et al., 2009), also known as, faceless
recognition (Oh et al., 2016). A pair-wise constraint is when two masked faces
can be determined to belong to the same person by using the variations of
clothing, body shape, or other cues, across images of the same person, even
though the faces are still hidden. Other soft-biometric features in full-body
photos, such as age, gender, race, ethnicity, birthmarks, and tattoos, provide
further hints for re-identification (Oh et al., 2016); however, such features
are more complex to detect. In the same direction, obstructing the entire body
provides better protection than face-masking alone; however, this way still
results in higher unfavorable utility loss. For instance, facial expressions
and body actions become imperceivable in the surveillance footage. The body
silhouette and the gait can leak information, therefore, Agrawal et al. (2011)
(Agrawal and Narayanan, 2011) proposed slightly dilating or expanding the
silhouette area to be obfuscated to hide the gait information and further
increase the privacy.

Reconstruction-based Attacks. The same attacks applied to obfuscated images
apply to redacted videos. However, there is another attack type applicable to
videos, utilizing temporal dependencies between adjacent frames in the video
sequences. Cavedon et al. (2011) (Cavedon et al., 2011) developed a technique
that completely recovers the identity of a pixelized face in video streams
under certain conditions. When pixelization is applied to video sequences
depicting the same subject, there is a high probability that the pixelization
squares will change position with respect to the image background, hence
averaging different pixels at different times. Unlike super-resolution
methods, the technique aims to recover the image with its original quality.
The authors used a Maximum Posteriori to find an image such that, if shifted
and pixelized in multiple frames, it will produce a video sequence as close as
possible to the input video. Their method assumes that the same image can be
tracked in different frames at different positions. A similar attack described
in (Padilla-López et al., 2015) uses image inpainting for interpolating masked
parts in a given frame using information from adjacent video frames.

The primary solution to this type of attack is to utilize more precise
detection mechanisms, perhaps by combining object detection, segmentation, and
tracking algorithms. Chen et al. (2008) (Chen et al., 2009) proposed a bi-
directional tracking algorithm that combines background subtraction, face
detection, and face tracking. Manual or automatic screening of the redacted
video frames can reduce missed face detections. Once a face is detected, it is
simultaneously tracked in both previous and subsequent frames to locate any
missed (usually non-frontal) faces. Background subtraction is used to segment
the foreground, containing people or other moving objects, from the
background. Furthermore, for face images, color-based detection can be
combined with other forms of detection to achieve better results (Chen et al.,
2018). Another countermeasure includes applying general obfuscation (i.e.,
full-frame obfuscation), but it is not favorable since it degrades the utility
of the video.

###  3.3. Discussion

Applications. Video de-identification is required in scenarios such as public
video publishing to protect, among others, the privacy of the individuals who
appear involuntarily in the videos. Furthermore, in public surveillance
videos, bypassers are captured mainly to record any behavioral anomalies;
their actions are relevant but not their identities in real-time. The need for
low-cost surveillance has driven the concept of privacy by design into smart
camera devices, so specific objects are never stored or transmitted, and
complex processing (such as natural face replacement) is moving towards edge
computing (Sah et al., 2017). Also, drones are now used for surveillance since
they can stealthily access a broader range of locations and closely track the
object of interest (Ribaric et al., 2016; Bonetto et al., 2015).

Challenges and Open Research. Face de-identification in videos is more
challenging than in still images. The challenges are related to the complexity
of facial detection and localization in video streams. Although face detection
in varying poses and profiles, illumination conditions, and occlusions have
been studied, the problem remains a known challenge. Similar to still images,
for an identity to be protected, multimodal de-identification must be
considered. Sensitive personal identifiers are of many diverse forms (e.g.,
skin, logos, location information such as house numbers, storefront signs,
street signs, and graffiti); therefore, fully automating the redaction process
still requires further research. Skin color-based segmentation still faces
limitations since skin occurs in many tones. Optical character recognition in
outdoor scenes is also limited by suboptimal conditions, such as poor lighting
and geometric perspective.

Key Properties. There are a few more considerations when selecting an
obfuscation method: real-time processing, low computational overhead,
reversibility, and utility-privacy trade-off in hiding identity information
while preserving behavior-related information (gait, actions, facial
gestures). Surveillance and monitoring systems that require real-time de-
identification should not use heavy processing methods such as inpainting,
encryption, and face replacement. Ad-hoc and simple transformations such as
blocking, masking, pixelization, and blurring techniques are more commonly
used in real-time scenarios, even though they provide lower protection.
Nonetheless, they are also not reversible and severely reduce the utility of
the video such that user actions and gait are not readily perceivable (Ivasic-
Kos et al., 2014). Some works recommend using other simple but utility-
preserving filters such as cartooning, edge detection, style transfer, false
colors, warping and morphing, where only the last three can be reversed if the
transformation parameters are known. When secure processing is required, the
more advanced transformations discussed previously should be used, with the
caveat that only encryption and scrambling are reversible.

Review of Methods. In Table 7, we list examples of real-time obfuscation
methods applicable to video surveillance scenarios. We also compare the
reversibility and the computational complexity of the methods—this latter
feature expressed in a qualitative appreciation of the required time. As
shown, video obfuscation follows the same protection techniques used in still
images. The exception is that the obfuscation systems would be slightly
modified to ensure the efficiency and speed of object detection, tracking, and
object denaturing.

Table 7. Examples of Real-time Video Obfuscation in the Literature

Method |  Example Time | Reversibility |
---|---|---|---
| Pixelization
---
Blurring (Body)
Masking
(Agrawal and Narayanan, 2011; Yu et al., 2008) | Fast | No
Cartooning | (Erdélyi et al., 2014) | Fast | No
Edge | (Chen et al., 2009) | Fast | No
Style transfer | (Brkić et al., 2016, 2016) | Slow | No
False colors | (Çiftçi et al., 2015) | Fast | Yes
Warping | (Korshunov and Ebrahimi, 2013b) | Fast | No
Face morphing | (Korshunov and Ebrahimi, 2013a) | Slow | No
Face replacement | (Yu et al., 2008; Gafni et al., 2019) | Slow | No
Scrambling | (Dufaux and Ebrahimi, 2008; Ruchaud and Dugelay, 2016) | Fast | Yes
Avatars (Body) | (Blažević et al., 2015) | Slow | No

##  4\. Conclusion

In this paper, we have provided an up-to-date and thorough review of the major
solutions for visual multimedia sanitization—the process of distorting,
replacing, or removing specific sensitive content in images and video. In
particular, we discuss current applications and constraints of existing visual
obfuscation methods, identify present challenges, and highlight possible
research directions. We have also shown that the discussed obfuscation methods
could be subject to various recognition, reconstruction, and inference
attacks—pointing to related trends and research directions to address the
cited challenges.

In conclusion, the quest that characterizes this research domain is to find an
optimal privacy-utility balance for each obfuscation method although, as shown
in this paper, there is not yet a common framework for evaluating the utility
and privacy of multimedia sanitization systems. Our contribution provides a
systematization of the filed, a highlight on the current trend and
technologies, as well as a few research directions to have sanitization
techniques meet the objective of providing the desired level of security and
privacy, while preserving the needed level of utility.

###### Acknowledgements.

This publication was made possible by an award [GSRA7-2-0527-20101] from Qatar
National Research Fund (a member of Qatar Foundation). This publication was
also partially supported by awards NPRP-S-11-0109-180242 “Extending Blockchain
Technology – a Novel Paradigm and its Applications to Cybersecurity and
Fintech” from the QNRF-Qatar National Research Fund, a member of The Qatar
Foundation, and NATO Science for Peace and Security Programme - MYP G5828
project. The contents herein are solely the responsibility of the authors.

## References

  * (1)
  * Agrawal and Narayanan (2011) Prachi Agrawal and PJ Narayanan. 2011.  Person de-identification in videos.  _IEEE Transactions on Circuits and Systems for Video Technology_ 21, 3 (2011), 299–310.
  * Akhtar et al. (2018) Naveed Akhtar, Jian Liu, and Ajmal Mian. 2018.  Defense Against Universal Adversarial Perturbations. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3389–3398.  https://doi.org/10.1109/CVPR.2018.00357
  * Anwar et al. (2020) Saeed Anwar, Salman Khan, and Nick Barnes. 2020.  A deep journey into super-resolution: A survey.  _ACM Computing Surveys (CSUR)_ 53, 3 (2020), 1–34.
  * Asghar et al. (2019) Mamoona N Asghar, Nadia Kanwal, Brian Lee, Martin Fleury, Marco Herbst, and Yuansong Qiao. 2019\.  Visual surveillance within the EU General Data Protection Regulation: A technology perspective.  _IEEE Access_ 7 (2019), 111709–111726.
  * Barnes et al. (2009) Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009.  PatchMatch: A randomized correspondence algorithm for structural image editing.  _ACM Trans. Graph._ 28, 3 (2009), 24.
  * Blažević et al. (2015) Martin Blažević, Karla Brkić, and Tomislav Hrkać. 2015.  Towards reversible de-identification in video sequences using 3d avatars and steganography.  _arXiv preprint arXiv:1510.04861_ (2015).
  * Bonetto et al. (2015) Margherita Bonetto, Pavel Korshunov, Giovanni Ramponi, and Touradj Ebrahimi. 2015.  Privacy in mini-drone based video surveillance. In _2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , Vol. 04. 1–6.  https://doi.org/10.1109/FG.2015.7285023
  * Brkić et al. (2017) Karla Brkić, Tomislav Hrkać, Zoran Kalafatić, and Ivan Sikirić. 2017.  Face, hairstyle and clothing colour de-identification in video sequences.  _IET Signal Processing_ 11, 9 (2017), 1062–1068.
  * Brkić et al. (2016) Karla Brkić, Ivan Sikirić, Tomislav Hrkać, and Zoran Kalafatić. 2016.  De-identifying people in videos using neural art. In _2016 Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA)_. IEEE, 1–6.
  * Brkic et al. (2017) Karla Brkic, Ivan Sikiric, Tomislav Hrkac, and Zoran Kalafatic. 2017.  I know that person: Generative full body and face de-identification of people in images. In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 1319–1328.
  * Brkić et al. (2016) Karla Brkić, Tomislav Hrkać, Ivan Sikirić, and Zoran Kalafatić. 2016.  Towards neural art-based face de-identification in video data. In _2016 First International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE)_. 1–5.  https://doi.org/10.1109/SPLIM.2016.7528406
  * Bursic et al. (2021) Sathya Bursic, Alessandro D’Amelio, Marco Granato, Giuliano Grossi, and Raffaella Lanzarotti. 2021\.  A Quantitative Evaluation Framework of Video De-Identification Methods. In _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 6089–6095.
  * Cavedon et al. (2011) Ludovico Cavedon, Luca Foschini, and Giovanni Vigna. 2011\.  Getting the Face Behind the Squares: Reconstructing Pixelized Video Streams.. In _WOOT_. 37–45.
  * Chaubey et al. (2020) Ashutosh Chaubey, Nikhil Agrawal, Kavya Barnwal, Keerat K Guliani, and Pramod Mehta. 2020\.  Universal adversarial perturbations: A survey.  _arXiv preprint arXiv:2005.08087_ (2020).
  * Chen and Nahrstedt (2019) Bo Chen and Klara Nahrstedt. 2019.  FIS: Facial Information Segmentation for Video Redaction. In _2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_. 360–365.
  * Chen et al. (2018) Bo Chen, Klara Nahrstedt, and Carl Gunter. 2018.  Response: Real-time, secure, and privacy-aware video redaction system. In _Proceedings of the 15th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services_. 39–48.
  * Chen et al. (2009) Datong Chen, Yi Chang, Rong Yan, and Jie Yang. 2009\.  Protecting personal identification in video.  In _Protecting Privacy in Video Surveillance_. Springer, 115–128.
  * Cheung et al. (2009) S-CS Cheung, M Vijay Venkatesh, Jithendra K Paruchuri, Jian Zhao, and Thinh Nguyen. 2009\.  Protecting and managing privacy information in video surveillance systems.  In _Protecting Privacy in Video Surveillance_. Springer, 11–33.
  * Çiftçi et al. (2017) Serdar Çiftçi, Ahmet Oğuz Akyüz, and Touradj Ebrahimi. 2017.  A reliable and reversible image privacy protection based on false colors.  _IEEE Transactions on multimedia_ 20, 1 (2017), 68–81.
  * Çiftçi et al. (2015) Serdar Çiftçi, Pavel Korshunov, Ahmet Oğuz Akyüz, and Touradj Ebrahimi. 2015\.  Using false colors to protect visual privacy of sensitive content. In _Human Vision and Electronic Imaging Xx_ , Vol. 9394. International Society for Optics and Photonics, 93941L.
  * Croft et al. (2019) William L Croft, Jörg-Rüdiger Sack, and Wei Shi. 2019.  Differentially Private Obfuscation of Facial Images. In _International Cross-Domain Conference for Machine Learning and Knowledge Extraction_. Springer, 229–249.
  * Deb et al. (2019) Debayan Deb, Jianbang Zhang, and Anil K Jain. 2019\.  Advfaces: Adversarial face synthesis. In _2020 IEEE International Joint Conference on Biometrics (IJCB)_. 1–10.
  * Du et al. (2014) Liang Du, Meng Yi, Erik Blasch, and Haibin Ling. 2014\.  GARP-face: Balancing privacy protection and utility preservation in face de-identification. In _IEEE International Joint Conference on Biometrics_. 1–8.
  * Dufaux and Ebrahimi (2008) Frederic Dufaux and Touradj Ebrahimi. 2008.  Scrambling for privacy protection in video surveillance systems.  _IEEE Transactions on Circuits and Systems for Video Technology_ 18, 8 (2008), 1168–1174.
  * Dwork (2008) Cynthia Dwork. 2008\.  Differential Privacy: A Survey of Results. In _Theory and Applications of Models of Computation_ , Manindra Agrawal, Dingzhu Du, Zhenhua Duan, and Angsheng Li (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1–19.
  * Emeršič et al. (2017) Žiga Emeršič, Vitomir Štruc, and Peter Peer. 2017\.  Ear recognition: More than a survey.  _Neurocomputing_ 255 (2017), 26–39.
  * Erdélyi et al. (2014) Adám Erdélyi, Tibor Barát, Patrick Valet, Thomas Winkler, and Bernhard Rinner. 2014\.  Adaptive cartooning for privacy protection in camera networks. In _2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. 44–49.
  * Fan (2018) Liyue Fan. 2018\.  Image pixelization with differential privacy. In _IFIP Annual Conference on Data and Applications Security and Privacy_. Springer, 148–162.
  * Fan (2019) Liyue Fan. 2019\.  Practical Image Obfuscation with Provable Privacy. In _2019 IEEE International Conference on Multimedia and Expo (ICME)_. 784–789.  https://doi.org/10.1109/ICME.2019.00140
  * Farrugia (2014) Reuben A Farrugia. 2014\.  Reversible De-Identification for lossless image compression using Reversible Watermarking. In _2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_. 1258–1263.
  * Gafni et al. (2019) Oran Gafni, Lior Wolf, and Yaniv Taigman. 2019.  Live face de-identification in video. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9378–9387.
  * Gross et al. (2005) Ralph Gross, Edoardo Airoldi, Bradley Malin, and Latanya Sweeney. 2005.  Integrating utility into face de-identification. In _International Workshop on Privacy Enhancing Technologies_. Springer, 227–242.
  * Gross et al. (2006) Ralph Gross, Latanya Sweeney, Fernando De la Torre, and Simon Baker. 2006.  Model-based face de-identification. In _2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’06)_. IEEE, 161–161.
  * Guo et al. (2017) Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. 2017.  Countering adversarial images using input transformations.  _arXiv preprint arXiv:1711.00117_ (2017).
  * Hasan et al. (2018) Rakibul Hasan, Eman Hassan, Yifang Li, Kelly Caine, David J Crandall, Roberto Hoyle, and Apu Kapadia. 2018.  Viewer experience of obscuring scene elements in photos to enhance privacy. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Hasan et al. (2019) Rakibul Hasan, Yifang Li, Eman Hassan, Kelly Caine, David J Crandall, Roberto Hoyle, and Apu Kapadia. 2019.  Can privacy be satisfying? on improving viewer satisfaction for privacy-enhanced photos using aesthetic transforms. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Hasan et al. (2017) Rakibul Hasan, Patrick Shaffer, David Crandall, Eman T Apu Kapadia, et al. 2017\.  Cartooning for enhanced privacy in lifelogging and streaming videos. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_. 29–38.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Early methods for detecting adversarial images.  _arXiv preprint arXiv:1608.00530_ (2016).
  * Hill et al. (2016) Steven Hill, Zhimin Zhou, Lawrence Saul, and Hovav Shacham. 2016\.  On the (in) effectiveness of mosaicing and blurring as tools for document redaction.  _Proceedings on Privacy Enhancing Technologies_ 2016, 4 (2016), 403–417.
  * Hrkać et al. (2016) Tomislav Hrkać, Karla Brkić, Slobodan Ribarić, and Darijan Marčetić. 2016\.  Deep learning architectures for tattoo detection and de-identification. In _2016 First International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE)_. IEEE, 1–5.
  * Hukkelås et al. (2019) Håkon Hukkelås, Rudolf Mester, and Frank Lindseth. 2019\.  Deepprivacy: A generative adversarial network for face anonymization. In _International Symposium on Visual Computing_. Springer, 565–578.
  * Ivasic-Kos et al. (2014) Marina Ivasic-Kos, Alexandros Iosifidis, Anastasios Tefas, and Ioannis Pitas. 2014.  Person de-identification in activity videos. In _2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_. IEEE, 1294–1299.
  * Jiddah and Yurtkan (2018) Salman Mohammed Jiddah and Kamil Yurtkan. 2018.  Fusion of geometric and texture features for ear recognition. In _2018 2nd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT)_. IEEE, 1–5.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4401–4410.
  * Kim and Lee (2014) Kyoung-nam Kim and MyounJae Lee. 2014.  Image obfuscation in the user-friendly sensitive area with the use of a sensor for smart devices and image processing techniques.  _International Journal of Distributed Sensor Networks_ 10, 5 (2014), 797353\.
  * Kim and Yang (2020) Taehoon Kim and Jihoon Yang. 2020.  Selective Feature Anonymization for Privacy-Preserving Image Data Publishing.  _Electronics_ 9, 5 (2020), 874.
  * Koh et al. (2021) Jaihyun Koh, Jangho Lee, and Sungroh Yoon. 2021.  Single-image deblurring with neural networks: A comparative survey.  _Computer Vision and Image Understanding_ 203 (2021), 103–134.
  * Korshunov et al. (2012) Pavel Korshunov, Claudia Araimo, Francesca De Simone, Carmelo Velardo, J-L Dugelay, and Touradj Ebrahimi. 2012.  Subjective study of privacy filters in video surveillance. In _2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP)_. 378–382.
  * Korshunov and Ebrahimi (2013a) Pavel Korshunov and Touradj Ebrahimi. 2013a.  Using face morphing to protect privacy. In _2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance_. 208–213.
  * Korshunov and Ebrahimi (2013b) Pavel Korshunov and Touradj Ebrahimi. 2013b.  Using warping for privacy protection in video surveillance. In _2013 18th International Conference on Digital Signal Processing (DSP)_. 1–6.  https://doi.org/10.1109/ICDSP.2013.6622791
  * Korshunov and Ebrahimi (2014) Pavel Korshunov and Touradj Ebrahimi. 2014.  Towards optimal distortion-based visual privacy filters. In _2014 IEEE International Conference on Image Processing (ICIP)_. 6051–6055.
  * Kumar et al. (2019) Ashu Kumar, Amandeep Kaur, and Munish Kumar. 2019\.  Face detection techniques: a review.  _Artificial Intelligence Review_ 52, 2 (2019), 927–948.
  * Kupyn et al. (2018) Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. 2018\.  Deblurgan: Blind motion deblurring using conditional adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 8183–8192.
  * Larson et al. (2018) Martha Larson, Zhuoran Liu, SFB Brugman, and Zhengyu Zhao. 2018\.  Pixel Privacy. Increasing Image Appeal while Blocking Automatic Inference of Sensitive Scene Information. In _Proceedings of the MediaEval 2018 Workshop_. Sophia Antipolis, Frankrijk:[Sn], 29–31.
  * Le et al. (2020) Minh-Ha Le, Md Sakib Nizam Khan, Georgia Tsaloli, Niklas Carlsson, and Sonja Buchegger. 2020.  AnonFACES: Anonymizing Faces Adjusted to Constraints on Efficacy and Security. In _Proceedings of the 19th Workshop on Privacy in the Electronic Society_. 87–100.
  * Ledig et al. (2017) Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017\.  Photo-realistic single image super-resolution using a generative adversarial network. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4681–4690.
  * Li et al. (2019) Chau Yi Li, Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, Riccardo Mazzon, and Andrea Cavallaro. 2019\.  Scene privacy protection. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 2502–2506.
  * Li and Lin (2019) Tao Li and Lei Lin. 2019\.  Anonymousnet: Natural face de-identification with measurable privacy. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 0–0.
  * Li and Lyu (2019) Yuezun Li and Siwei Lyu. 2019.  De-identification without losing faces. In _Proceedings of the ACM Workshop on Information Hiding and Multimedia Security_. 83–88.
  * Li et al. (2017) Yifang Li, Nishant Vishwamitra, Bart P Knijnenburg, Hongxin Hu, and Kelly Caine. 2017.  Effectiveness and users’ experience of obfuscation as a privacy-enhancing technology for sharing photos.  _Proceedings of the ACM on Human-Computer Interaction_ 1, CSCW (2017), 1–24.
  * Liu et al. (2020) Chi Liu, Tianqing Zhu, Jun Zhang, and Wanlei Zhou. 2020\.  Privacy Intelligence: A Survey on Image Sharing on Online Social Networks.  _arXiv preprint arXiv:2008.12199_ (2020).
  * Lugini et al. (2014) Luca Lugini, Emanuela Marasco, Bojan Cukic, and Jeremy Dawson. 2014.  Removing gender signature from fingerprints. In _2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_. IEEE, 1283–1287.
  * Mahajan et al. (2017) Sachit Mahajan, Ling-Jyh Chen, and Tzu-Chieh Tsai. 2017\.  Swapitup: A face swap application for privacy protection. In _2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA)_. 46–50.
  * Marčetic et al. (2014) Darijan Marčetic, Slobodan Ribarić, Vitomir Štruc, and Nikola Pavešić. 2014.  An experimental tattoo de-identification system for privacy protection in still images. In _2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_. 1288–1293.  https://doi.org/10.1109/MIPRO.2014.6859766
  * McPherson et al. (2016) Richard McPherson, Reza Shokri, and Vitaly Shmatikov. 2016\.  Defeating image obfuscation with deep learning.  _arXiv preprint arXiv:1609.00408_ (2016).
  * Meden et al. (2018) Blaž Meden, Žiga Emeršič, Vitomir Štruc, and Peter Peer. 2018\.  k-Same-Net: k-Anonymity with generative deep neural networks for face deidentification.  _Entropy_ 20, 1 (2018), 60.
  * Meng and Sun (2014) Lily Meng and Zongji Sun. 2014.  Face De-identification with perfect privacy protection. In _2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_. IEEE, 1234–1239.
  * Mondal (2018) Bhaskar Mondal. 2018\.  Cryptographic image Scrambling techniques.  _Cryptographic and Information Security Approaches for Images and Videos_ (2018), 37\.
  * Mondal et al. (2017) Bhaskar Mondal, Neel Biswas, and Tarni Mandal. 2017\.  A Comparative study on Cryptographic Image Scrambling.. In _RICE_. 261–268.
  * Nakashima et al. (2015) Yuta Nakashima, Tatsuya Koyama, Naokazu Yokoya, and Noboru Babaguchi. 2015.  Facial expression preserving privacy protection using image melding. In _2015 IEEE International Conference on Multimedia and Expo (ICME)_. 1–6.
  * Nambiar et al. (2019) Athira Nambiar, Alexandre Bernardino, and Jacinto C Nascimento. 2019\.  Gait-based person re-identification: A survey.  _ACM Computing Surveys (CSUR)_ 52, 2 (2019), 1–34.
  * Newton et al. (2005) Elaine M Newton, Latanya Sweeney, and Bradley Malin. 2005\.  Preserving privacy by de-identifying face images.  _IEEE transactions on Knowledge and Data Engineering_ 17, 2 (2005), 232–243.
  * Oh et al. (2016) Seong Joon Oh, Rodrigo Benenson, Mario Fritz, and Bernt Schiele. 2016.  Faceless person recognition: Privacy implications in social media. In _European Conference on Computer Vision_. Springer, 19–35.
  * Oh et al. (2017) Seong Joon Oh, Mario Fritz, and Bernt Schiele. 2017\.  Adversarial image perturbation for privacy protection a game theory perspective. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 1491–1500.
  * Othman and Ross (2014) Asem Othman and Arun Ross. 2014.  Privacy of facial soft biometrics: Suppressing gender but retaining identity. In _European Conference on Computer Vision_. Springer, 682–696.
  * Padilla-López et al. (2015) José Ramón Padilla-López, Alexandros Andre Chaaraoui, and Francisco Flórez-Revuelta. 2015\.  Visual privacy protection methods: A survey.  _Expert Systems with Applications_ 42, 9 (2015), 4177–4195.
  * Pan et al. (2019) Yi-Lun Pan, Min-Jhih Haung, Kuo-Teng Ding, Ja-Ling Wu, and Jyh-Shing Jang. 2019.  K-Same-Siamese-GAN: K-Same Algorithm with Generative Adversarial Network for Facial Image De-identification with Hyperparameter Tuning and Mixed Precision Training. In _2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. 1–8.
  * Poller et al. (2012) Andreas Poller, Martin Steinebach, and Huajian Liu. 2012\.  Robust image obfuscation for privacy protection in web 2.0 applications. In _Media Watermarking, Security, and Forensics 2012_ , Vol. 8303. International Society for Optics and Photonics, 830304.
  * Poyser et al. (2020) Matt Poyser, Amir Atapour-Abarghouei, and Toby P Breckon. 2020\.  On the Impact of Lossy Image and Video Compression on the Performance of Deep Convolutional Neural Network Architectures.  _arXiv preprint arXiv:2007.14314_ (2020).
  * Prinosil (2018) Jiri Prinosil. 2018\.  Clothing Color Based De-Identification. In _2018 41st International Conference on Telecommunications and Signal Processing (TSP)_. IEEE, 1–5.
  * Prinosil et al. (2015) Jiri Prinosil, Ales Krupka, Kamil Riha, Malay Kishore Dutta, and Anushikha Singh. 2015.  Automatic hair color de-identification. In _2015 International Conference on Green Computing and Internet of Things (ICGCIoT)_. IEEE, 732–736.
  * Qin et al. (2017) Siyang Qin, Seongdo Kim, and Roberto Manduchi. 2017.  Automatic skin and hair masking using fully convolutional networks. In _2017 IEEE International Conference on Multimedia and Expo (ICME)_. 103–108.
  * Ra et al. (2013) Moo-Ryong Ra, Ramesh Govindan, and Antonio Ortega. 2013\.  P3: Toward privacy-preserving photo sharing. In _Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation_. 515–528.
  * Rajabi et al. (2021) Arezoo Rajabi, Rakesh B Bobba, Mike Rosulek, Charles V Wright, and Wu-chi Feng. 2021.  On the (Im) Practicality of Adversarial Perturbation for Image Privacy.  _Proceedings on Privacy Enhancing Technologies_ 2021, 1 (2021), 85–106.
  * Rakhmawati et al. (2018) Lusia Rakhmawati, Wirawan, and Suwadi. 2018\.  Image Privacy Protection Techniques: A Survey. In _TENCON 2018 - 2018 IEEE Region 10 Conference_. 0076–0080.  https://doi.org/10.1109/TENCON.2018.8650339
  * Raval et al. (2017) Nisarg Raval, Ashwin Machanavajjhala, and Landon P Cox. 2017\.  Protecting visual secrets using adversarial nets. In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 1329–1332.
  * Ren et al. (2018) Zhongzheng Ren, Yong Jae Lee, and Michael S Ryoo. 2018\.  Learning to anonymize faces for privacy preserving action detection. In _Proceedings of the european conference on computer vision (ECCV)_. 620–636.
  * Ribaric et al. (2016) Slobodan Ribaric, Aladdin Ariyaeeinia, and Nikola Pavesic. 2016\.  De-identification for privacy protection in multimedia content: A survey.  _Signal Processing: Image Communication_ 47 (2016), 131–151.
  * Ribaric and Pavesic (2015) Slobodan Ribaric and Nikola Pavesic. 2015.  An overview of face de-identification in still images and videos. In _2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , Vol. 4. 1–6.
  * Ruchaud and Dugelay (2016) Natacha Ruchaud and Jean-Luc Dugelay. 2016.  Privacy protecting, intelligibility preserving video surveillance. In _2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_. 1–6.
  * Sah et al. (2017) Shagan Sah, Ameya Shringi, Raymond Ptucha, Aaron M Burry, and Robert P Loce. 2017.  Video redaction: a survey and comparison of enabling technologies.  _Journal of Electronic Imaging_ 26, 5 (2017), 051406\.
  * Sanchez-Matilla et al. (2020) Ricardo Sanchez-Matilla, Chau Yi Li, Ali Shahin Shamsabadi, Riccardo Mazzon, and Andrea Cavallaro. 2020\.  Exploiting vulnerabilities of deep neural networks for privacy protection.  _IEEE Transactions on Multimedia_ 22, 7 (2020), 1862–1873.
  * Scherhag et al. (2019) Ulrich Scherhag, Christian Rathgeb, Johannes Merkle, Ralph Breithaupt, and Christoph Busch. 2019\.  Face recognition systems under morphing attacks: A survey.  _IEEE Access_ 7 (2019), 23012–23026.
  * Seibold et al. (2017) Clemens Seibold, Wojciech Samek, Anna Hilsmann, and Peter Eisert. 2017.  Detection of face morphing attacks by deep learning. In _International Workshop on Digital Watermarking_. Springer, 107–120.
  * Sharma and Chen (2018) Sagar Sharma and Keke Chen. 2018.  Image disguising for privacy-preserving deep learning. In _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_. 2291–2293.
  * Sun et al. (2018) Qianru Sun, Liqian Ma, Seong Joon Oh, Luc Van Gool, Bernt Schiele, and Mario Fritz. 2018\.  Natural and effective obfuscation by head inpainting. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 5050–5059.