# Human-Imperceptible Identification with Learnable Lensless Imaging

Thuong Nguyen Canh, Trung Thanh Ngo, and Hajime Nagahara

###### Abstract

Lensless imaging protects visual privacy by capturing heavily blurred images
that are imperceptible for humans to recognize the subject but contain enough
information for machines to infer information. Unfortunately, protecting
visual privacy comes with a reduction in recognition accuracy and vice versa.
We propose a learnable lensless imaging framework that protects visual privacy
while maintaining recognition accuracy. To make captured images imperceptible
to humans, we designed several loss functions based on total variation,
invertibility, and the restricted isometry property. We studied the effect of
privacy protection with blurriness on the identification of personal identity
via a quantitative method based on a subjective evaluation. Moreover, we
validate our simulation by implementing a hardware realization of lensless
imaging with photo-lithographically printed masks.

###### Index Terms:

Lensless imaging, deep learning, compressive sensing, convolution neural
network, privacy preserving.

##  I Introduction

Visual privacy is the right to limit the collection and use of visual
information related to personal identity. With the proliferation of cameras in
smartphones, wearable devices, laptops, and surveillance equipment, it is
increasingly easy to photograph someone without being noticed, violating their
visual privacy. At the same time, visual appearance, such as facial data, has
become the most commonly used feature in biometric authentication for
accessing private information [1, 2], and its usage has significantly
increased in recent years. Additionally, advanced virtual and augmented
reality headsets are equipped with multiple cameras, which are most likely
used in work and personal environments [3] containing sensitive visual
information. Alongside these hardware developments, artificial intelligence
has advanced accuracy in many computer vision tasks, raising the question of
privacy. In fact, recent research [4] identified privacy protection as one of
the most important ethical principles of artificial intelligence. Therefore,
imaging systems should consider not only its functionality but also its
ability to protect visual privacy. Fortunately, computer vision tasks can be
performed without knowledge of subject identity [5, 6, 7].

On the other hand, visual privacy can be classified as the protection of
sensitive visual information against machine vision and/or human visual
systems. For example, visual privacy against machine vision using adversarial
samples, a sub-topic of adversarial machine learning [8], aims to fool the
targeted machine vision system [9]. For example, adversarial images usually
succeed in fooling machine recognition via the addition of noise, objects, or
via cartoonization, but often fail to fool human vision [10]. It should be
noted that we pursue a completely different direction. Our goal is to deceive
human vision by capturing a human-imperceptible image without preventing
machine recognition. Therefore, we can enable computer vision applications
without compromising visual privacy.

In general, it is preferable to implement a high degree of visual protection
at the physical or sensor level to prevent attacks during information
transmission and to reduce the vulnerability to software hacks. One promising
method is secure imaging. Secure imaging is an active research area that
studies image capture using secure formats, such that the captured images are
visually protected. Unlike many software-based approaches that are vulnerable
to malware [11] [12], the output of an image sensor is secure and difficult to
hack, thus enabling hardware-level privacy protection. Previous studies have
applied several methods for developing secure cameras [13], [14]. Among them,
lensless imaging [15] is the most common method. Lensless imaging processes
incoming light by replacing the lens system with a coded mask [15] and
reconstructs a visually understandable image from the captured measurements.
Researchers have realized lensless imaging based on compressive sensing (CS)
technology, developing systems such as a single-pixel camera with a directed
micromirror [16], a single-pixel lensless imaging system with a coded liquid
crystal display (LCD) [17], [18]. These systems capture a very blurry image,
which represents a simple way of protecting visual privacy but paid no
attention to the trade-off between visual privacy and recognition accuracy
[19].

In this work, we make the following contributions:

  * •

We propose an end-to-end framework for learning the coded pattern of a human-
imperceptible recognizer;

  * •

We propose similarity, total variation (TV), invertibility, and restricted
isometry property (RIP) losses to protect visual privacy against human vision;

  * •

We evaluate visual privacy using the blurriness of captured measurements, a
RIP score, and perceptual experiments on face identification;

  * •

We implement a lensless imaging system with various printed masks.

The rest of this paper is organized as follows. We review related work in
Section II. Section III introduces our learnable lensless imaging system for
recognition, and Section IV presents various loss functions for preserving
visual privacy. We describe the perceptual evaluation experiment in Section V
alongside a hardware realization of our system, and we present our conclusions
in Section VI.

##  II Related Work

To protect visual privacy, together with cartoonization, blanking, scrambling,
blur the sensitive region is a common practice to protect visual privacy again
human visual system in both academic [11], [20], [21] and industry (i.e.
Google map, etc.). In general, encoder side protection is preferred to prevent
transmission attach and could be further classified into the two approaches
discussed below.

###  II-A Software-based Approach

Software-based approaches [22], [21] process the captured image by applying
cartoonization, blanking, and scrambling sensitive regions [11], [20] before
performing high-level computer vision tasks. However, private visual
information remains available to software attacks [1], because the sensitive
regions still exist in the image captured by a regular camera before
processing. In addition, a single failure in detecting sensitive regions may
reveal the identity of objects. CS theory has been adopted to capture secure
images [23], [20], [24] with a stronger level of protection. That is the whole
is mage is imperceptible, thus avoiding detection failure. Recent studies have
shown that it is possible to understand the context from CS measurements with
[25] or without reconstruction [23], [26].

###  II-B Hardware-based Approach

For protection at the hardware level, researchers have used thermal images
[14], [27], defocused images [13], and low-resolution images [7], [28]. Tan et
al. [29] achieved remarkable accuracy using the lensless FlatCam system [17]
for face detection and verification tasks. Canh et al. [30] further developed
FlatCam for visual privacy applications by protecting sensitive regions in the
measurement domain. However, both methods require an initial reconstruction
stage, which remains vulnerable to software attacks. A previous study has
investigated reconstruction-free inference for lensless action recognition
[31]. The authors of this study captured multiple lensless measurements and
used phase correlation and translation motion features to recognize action
accuracy regardless of the coded patterns. Their method requires no
reconstruction, thus avoiding the training complexity associated with learning
multiple patterns. Tan et al. [19] further enhanced visual privacy protection
by considering additional analog processes such as quantization. While
previous work [31] has determined that measurements captured by lensless
imaging are generally unrecognizable to humans, the accuracy tradeoff between
human-based and machine-based recognition has received little or no attention
[19]. The extent to which lensless imaging impacts the human perception of
identity remains unknown, with a lack of research on perceptual and objective
quality assessment for recognition.

###  II-C Visual Privacy of Lensless Imaging

In general, the blurriness of lensless coded images hides visual information,
and therefore it has been assumed to protect visual privacy in lensless
imaging [29, 30]. Conversely, most results from the image quality assessment
field have focused on the quality of the displayed image, which is not
directly related to the recognition capability of blurry images. Previous
studies [32, 33] have evaluated the blurriness of images but have often
considered motion blur and out-of-focus blur, which are not associated with
recognition accuracy. Another concern is the reconstruction of a lensless
image can reveal visual information via a blind deconvolution [34]. Therefore,
reducing invertibility (i.e., the ability to reconstruct clear images from
blurry images) is an essential factor in protecting visual privacy. To address
this requirement, we use the restricted isometry property (RIP) [35, 36] to
measure the invertibility of lensless images.

Figure 1: Lensless imaging with (a) short distance and (b) long distance
between the image sensor and coded pattern.

Additionally, most previous work on visual privacy has used manually coded
patterns to capture blurry images but achieved poor recognition accuracy.
Although it is possible to learn an efficient pattern for high accuracy, this
also reduces the blurriness of captured images and thereby reduces visual
privacy. Moreover, the relationship between blurriness and visual privacy
protection has not been addressed. Our work subjectively evaluates the
recognition of lensless images and proposes a method to protect visual privacy
while maintaining high recognition accuracy. Researchers have simply
associated the blurriness of coded measurements with visual privacy protection
[30, 29]. However, how well lensless imaging can protect the visual privacy of
coded measurements for recognition is still unknown.

##  III Secure Lensless Imaging

###  III-A Lensless Imaging

Lensless imaging is an image capture technique without a complex lens system.
A coded pattern is used to modulate the incoming light with single-pixel [15]
or multiple-pixel [17] cameras. As camera sensors have become less expensive,
the latter approach has become more popular, because it enables single-shot
image capture without changing coded patterns. As shown in Fig. 1 (a)–(b),
given an image $x$ and a coded pattern $H$, the lensless measurement $y$ is
modeled by

$y=H*x+\eta,$ (1)

where $*$ is the convolution operator and $\eta$ represents the additive noise
source. When the distance $d_{1}$ between the image and the corresponding
coded pattern is decreased, the camera becomes thinner [17]. This factor also
limits the angle of the incoming light rays and thereby restricts the field of
view. At the same resolution of $H$, a larger $d_{1}$ leads to a blurrier
image because of the large kernel size, thereby providing better visual
privacy. We, therefore, adopt a large $d_{1}$ value, and we ignore noise and
diffraction for simplicity.

Fig. 3 shows examples of various fixed patterns (Pinhole, Full open and
random), our proposed (Learned pattern), and their corresponding captured
images. Subject identity is seen by the pinhole pattern, but not by the random
and fully open patterns. Lacking an objective metric to evaluate visual
privacy, previous work has relied on the blurriness metric as a simple privacy
indicator [37]. Unfortunately, blurriness [37] is not a satisfactory measure
of human imperceptibility: for example, higher blurriness nevertheless reveals
more information in the learned pattern of Fig. 3. There is a demand for a
better evaluation method. In later work, we associate the visual privacy of
lensless imaging with the ability to reconstruct an image from its
measurement. Therefore, we proposed approximated inverse RIP property as a
criterion in Section IV.

Figure 2: Proposed face recognition system with lensless imaging. Figure 3:
Visual comparison among different coded patterns with fixed (pinhole, full
open, random) and learned pattern. Numerical values above measured images and
corresponding patterns indicate [blurriness] [37] and (aperture ratio),
respectively. The aperture ratio is the proportion of “bright” (equal to 1)
pixels in the pattern.

###  III-B Lensless Imaging for Human-Imperceptible Identification

Our lensless imaging system for identification, illustrated in Fig. 2,
captures images using a lensless camera with a pattern $H$, and then sends
them to a recognizer without reconstruction. An early study [31] used hand-
crafted patterns, but the better performance was subsequently achieved with
learned patterns [38, 39]. Therefore, our approach involves joint learning of
both the $H$ pattern and the recognizer, while preserving human
imperceptibility and preventing reconstruction. It is straightforward to use
the identification loss $\mathcal{L}_{\mathsf{rec}}(\mathcal{R}(H*x),b)$ for
training this network, where $\mathcal{L}_{\mathsf{rec}}$ denotes the cross-
entropy loss with a recognizer $\mathcal{R}$, an input image $x$, and a label
$b$. The total training loss $\mathcal{L}$ is

$\mathcal{L}=\mathcal{L}_{\mathsf{rec}}(\mathcal{R}(H*x),b)+\alpha\mathcal{L}_{\mathsf{hi}},$
(2)

where $\mathcal{L}_{\mathsf{hi}}$ denotes human-imperceptible loss (further
detailed in the next section). We adopted the method of [38] to learn the
binary patterns $H$, and we used ResNet18 as an example for the recognizer.

###  III-C Scope of Proposed Framework

As the concept of privacy is very broad, only visual privacy is considered in
this work. In traditional imaging, privacy is protected at the software level,
which is vulnerable to software attacks. In the worst-case scenario, attackers
can gain access to the camera output and retrieve all available privacy
information. In contrast, lenselss imaging offers privacy protection at a
physical level. Even in the event of completely successful attacks, attackers
only gain access to the blurred measurement.

Our proposed lensless framework is able to fool both human and machine vision
to a certain extent. On one hand, the captured image is extremely blurry and
imperceptible to the human visual system. On the other hand, without knowing
the coded pattern, attacking the lensless measurement without knowing the
coded pattern results in poor reconstruction. In order to guess the coded
pattern, attackers must engage in a plain-attack scenario under which they
gain the ability to (1) manipulate the input and output of the camera, (2)
collect multiple images, and (3) perform training geared towards guessing the
coded pattern.

##  IV Human-Imperceptible Loss

This section describes loss functions that balance privacy and recognition
performance. We propose four methods in the form of the loss function to learn
the coded patterns. These methods are based on different components of the
imaging model and are motivated by theoretical considerations. We first
introduce measurement-based similarity loss, followed by loss based on coded
patterns in combination with TV, invertibility, and RIP.

###  IV-A Measurement-Based Human-Imperceptible Loss

In general, the loss function should be some objective quality metric that is
highly correlated with the results of the subjective evaluation described in
Section V-D and Fig. 6. Unfortunately, there are no existing visual privacy
metrics. To address this issue, we devised a protocol for assessing the impact
of blurriness on human perception of blurred images. A simpler approach may
involve maximization of the difference between input images and corresponding
captured images, however, this approach is unsatisfactory because many loss
functions (such as MSE and SSIM) are sensitive to pixel shifting. For example,
the learned patterns are shifted slightly upward in the learned pattern with
RIP loss of Fig. 4, to capture the forehead, eye, and nose.

####  IV-A1 Similarity Loss.

To overcome the above drawbacks, we propose an alternative approach that uses
images captured with a fully open aperture as the most visually secure images
(examples are shown in the measured image of the Full open pattern in Fig. 4).
Instead of maximizing the blurriness of captured images, we minimize the
difference between the learned measurement and the images captured with a
fully open aperture. We, therefore, regard the fully open aperture as the most
visually secure image, and we minimize:

$\mathcal{L}_{\mathsf{sim}}=\sum_{i}||H*x_{i}-{1}_{m}*x_{i}||^{2}_{2},$ (3)

where ${1}_{m}$ denotes the all-ones matrix, which is the coded pattern for
fully open aperture imaging, and $x_{i}$ denotes the $i^{\textrm{th}}$ image
in the training data. We use mean squared error for simplicity here, but other
loss functions (such as L1 norm) or perceptual metrics (such as SSIM) could
also be adopted.

Figure 4: Visual comparison across examples of learned lensless imaging for
various combinations of image size ($63\times 63,127\times 127$) and pattern
size ($32\times 32,64\times 64$). Numerical figures shown above measurement
images and corresponding patterns indicate [blurriness] $\uparrow$ [37] and
(aperture ratio) $\uparrow$, respectively. Ranking in terms of decreasing
revealed privacy is: fully open, random patterns, learned patterns with
constraints, and learned patterns without constraint.

###  IV-B Pattern-Based Human-Imperceptible Loss

This section presents various loss functions $\mathcal{L}_{\mathsf{hi}}$ as
constraints to learn the coded $H$ pattern.

####  IV-B1 Total Variation Loss

As shown in all learned patterns without constraint Fig. 4, in the absence of
constraints the learned patterns converged to a local or pinhole-like pattern.
As a result, measurements were convoluted from a small region of the images,
thus revealing visual information. Unfortunately, similarity loss could not
overcome this drawback. However, as the pattern converges locally, its TV also
becomes small. A high TV pattern combines distant pixels and reduces local
correlations. Therefore, the TV [40] of $H$ is maximized with:

$\mathcal{L}_{\mathsf{tv}}=-||\Delta_{x}H||_{1}-||\Delta_{y}H||_{1},$ (4)

where $\Delta_{x}$ and $\Delta_{y}$ represent horizontal and vertical gradient
operators, respectively. With TV loss, as can be seen from Fig. 4 the learned
patterns become more diverse and degrade visual information.

####  IV-B2 Invertibility Loss

TABLE I: Top 1 Accuracy (%) $\uparrow$ of various imaging setups with ResNet18. Patt. | Dataset | MS-Celeb | VGG-Face2 | CASIA
---|---|---|---|---
Image size | 63$\times$63 | 127$\times$127 | 63$\times$63 | 127$\times$127 | 63$\times$63 | 127$\times$127
Patt. size | 32$\times$32 | 32$\times$32 | 64$\times$64 | 32$\times$32 | 32$\times$32 | 64$\times$64 | 32$\times$32 | 32$\times$32 | 64$\times$64
Coded ratio | 1/4 | 1/16 | 1/4 | 1/4 | 1/16 | 1/4 | 1/4 | 1/16 | 1/4
Fix | Pinhole | 98.57 | 98.51 | 99.35 | 96.39 | 97.68 | 98.89 | 95.10 | 97.40 | 96.20
Full-open | 76.88 | 92.74 | 77.96 | 48.21 | 74.36 | 76.89 | 64.01 | 83.46 | 65.16
Random | 74.77 | 92.58 | 78.52 | 50.36 | 75.42 | 82.90 | 66.05 | 85.27 | 69.24
Learn | LwoC | 92.36 | 99.17 | 95.77 | 87.67 | 95.00 | 88.92 | 89.19 | 95.00 | 92.75
LwC-Sim | 93.12 | 99.02 | 91.11 | 86.54 | 94.18 | 87.11 | 89.31 | 94.23 | 90.89
LwC-TV | 93.23 | 99.04 | 96.18 | 86.37 | 94.16 | 91.63 | 89.77 | 94.10 | 90.38
LwC-Inv | 85.78 | 92.38 | 84.54 | 75.60 | 93.49 | 73.44 | 84.51 | 91.47 | 83.23
LwC-RIP | 91.12 | 95.18 | 93.20 | 78.62 | 89.53 | 81.31 | 88.80 | 92.67 | 86.68
Best results are in red bold, second best in bold.

Even though it is difficult to reconstruct an image from a lensless
measurement because of its poor invertibility [17, 30, 41], it is still
possible to do so. Therefore, visual privacy protection is not fully
preserved. Unfortunately, there has been limited research on invertibility for
lensless imaging. Yang et al. [42] proposed an invertibility regularization
algorithm to improve reconstruction performance. Regularization biases the
learned pattern towards a pinhole pattern. For a binary pattern $H$,
invertibility loss is simplified by maximizing the L1-norm of the learned
pattern. Unlike [42], we minimize the potential for reconstruction, and define
loss as the negative function:

$\mathcal{L}_{\mathsf{inv}}=-||H||_{1}.$ (5)

Minimizing $\mathcal{L}_{\mathsf{inv}}$ leads to a pattern with a large
aperture opening. The learned pattern makes the captured image blurry, and
difficult to reconstruct.

####  IV-B3 Restricted Isometry Property Loss

One obvious break in the visual privacy of lensless imaging is to reconstruct
the input image from lensless measurement. Therefore, visual privacy is
directly related to the ability to reconstruct lensless measurements. In this
regard, restricted isometry property (RIP) [35] is a well-known condition for
guaranteeing the reconstruction of sparse signals at a high probability. As in
[35], RIP condition is formalized as

$(1-\delta)||x||_{2}^{2}\leq||H*x||_{2}^{2}\leq(1+\delta)||x||^{2}_{2},$ (6)

The expression above states that “if a pattern $H$ preserves distance at small
$\delta$ for all $s$\- sparse signals $x$, then there is a high probability
that $x$ can be reconstructed with high quality”. The better in satisfying the
RIP condition of measurement, the higher chance to reconstruct the original
signal at high quality. The inverse is also true as, the less satisfying the
RIP condition is, the less guarantee of reconstruction of the signal. In our
goal, we would like to capture lensless measurements with poor RIP properties
to make it more difficult to reconstruct. Therefore, we would like to minimize
the RIP condition.

In order to do so, we simplified RIP condition by dividing all side in Eq. 6
by the signal energy as

$(1-\delta)\leq\frac{||H*x||_{2}^{2}}{||x||_{2}^{2}+\epsilon}\leq(1+\delta),$
(7)

where $\epsilon=10^{-10}$ is a small constraint added to avoid division by
zero. As the learned pattern $H$ in lensless imaging is a binary matrix,
measurement energy is always smaller than signal energy
$||H*x||_{2}^{2}\leq||x||_{2}^{2}$. Therefore, only left equation of Eq. 7 is
necessary as

$(1-\delta)\leq\frac{||H*x||_{2}^{2}}{||x||_{2}^{2}+\epsilon}.$ (8)

From Eq. 8, minimizing RIP condition of lensless imaging is equivalent to
maximize the isometric constraint $\delta$, or minimizing the ratio between
energy of measurement and signal. Unfortunately, the RIP condition is
nonlinear because it must be satisfied for all $s$-sparse signals, thus, it is
difficult to utilize RIP in a learning framework. We, therefore, approximate a
small RIP condition for only $n$ sample in each training batch by maximizing
the negative energy ratio as

$\mathcal{L}_{\mathsf{rip}}=-\sum_{i}\frac{||H*x_{i}||_{2}^{2}}{||x_{i}||_{2}^{2}+\epsilon}.$
(9)

By doing so, $\mathcal{L}_{\mathsf{rip}}$ can be calculated and integrated to
the neural training framework as a loss function.

It is also worth mentioning that, the curve of isometric constraint is used to
evaluate the visual privacy performance in Section V.

##  V Experimental Results

###  V-A Dataset and Training

In evaluating the proposed human-imperceptible recognition system, we set up a
facial identification system with various fields of view from three datasets:
VGG-Face2 [43], aligned Microsoft Celeb (MS-Celeb) [44], and CASIA [45]. For
each dataset, we selected the ten classes containing the largest number of
images, and then divided them into training and test sets with a ratio of
95:5. Images were converted to gray-scale and resized to the three different
specifications $\\{n,m\\}=\\{63\times 63,32\times 32\\},\\{127\times
127,32\times 32\\}$, and $\\{127\times 127,64\times 64\\}$, where $n,m$
indicate pattern size of the captured image and coded pattern. For training
augmentation, we used random cropping with a padding size of 4 and vertical
flipping (with approximately 6520 images per epoch).

Each network was associated with a loss function defined for a fixed or
learned coded pattern. Networks were trained from scratch with a stochastic
gradient descent optimizer. The initial learning rate was 0.2 and was reduced
to a fifth of its value every 100 epochs. We used 600 epochs for training, a
momentum of 0.9, and a weight decay of $5\times 10^{-4}$. The mini-batch size
was 128. The epoch with the best top-1 classification accuracy was selected as
the final solution. In the simulation results, we independently trained each
dataset.

Figure 5: (a) Invertibility performance with AUC-RIP $\downarrow$ and (b)
tradeoff between accuracy and RIP. Random and full open patterns achieved the
same AUC-RIP. Learned pattern with constraints LwC-Inv and LwC-RIP also shared
the same AUC-RIP, but LwC-RIP is superior because of its higher accuracy.

###  V-B Benchmark Methods

We evaluated lensless imaging with both fixed and learned patterns. For
lensless imaging with fixed patterns, we tested four identification systems
that used a pinhole pattern (indicated by Pinhole), a fully open pattern
(Full-open), and a random pattern. For lensless imaging with learned patterns,
different methods were trained without privacy constraints as LwoC. Finally,
methods that relied on human-imperceptible loss are named LwC-Sim, LwC-TV,
LwC-Inv, and LwC-RIP.

###  V-C Identification Results on Simulated Data

It is challenging to evaluate human recognition of images because there is
relatively little quantitative research on measuring the ability of the human
visual system to recognize objects. In general, it is more difficult for
humans to determine subject identity from images when they are blurred.
Therefore, we adopted a no-reference objective blur metric [37] to evaluate
the quality of visual privacy. Table I shows that the highest accuracy was
obtained for pinhole imaging. Lensless imaging with fully open and random
patterns resulted in a 20%–40% accuracy loss. When evaluated using human-
imperceptible identification (shown in the measured image of the Pinhole
pattern in Fig. 4), pinhole imaging revealed image details, while imaging with
fully open and random patterns did not. As expected, the learned patterns
improved recognition accuracy, yielding approximately a 5% loss in accuracy
compared with pinhole imaging, but failed to guarantee visually protected
measurements. As we expected, without any constraint, the learned pattern in
LwoC-woRec achieved the second-best recognition performance. Unfortunately, it
revealed more visual information. For instance, at a small coded ratio
$r=n/m$, LwoC-woRec revealed subject identity as shown in all measured images
Without Constraint in Fig. 4. Therefore, it is desirable to develop a method
to control the tradeoff between accuracy and privacy.

####  V-C1 Learned patterns and loss functions

The recognition accuracy of the constrained methods decreases in the following
order: LwC-Sim, LwC-TV, LwC-RIP, and LwC-Inv. However, similarity loss is
ineffective in protecting visual privacy. Additionally, blurriness is not a
good quality index. For example, Fig. 4 demonstrates that the degree of
blurriness estimated via invertibility loss (rows 2–4, column 7) was greater
than that estimated via RIP loss (rows 2–4, column 9), however, the
corresponding image reveals more visual information.

To better assess invertibility in a perceptually meaningful manner, we used
RIP as a quality metric. First, the RIP of matrix $H$ is represented as a RIP
curve for a given set of data. In a RIP curve, at a given $\delta\in[0,1]$, we
measure the percentage of pattern $H$ that satisfies the RIP condition [36].
To obtain an objective score, we calculate the area under the curve (AUC) of
the RIP curve, which is named AUC-RIP (see Fig. 5-a). The more difficult it is
to reconstruct the image from the captured measurement, the better visual
protection offered by pattern $H$ against reconstruction. As a rule of thumb,
a smaller AUC-RIP is preferred. LwoC and LwC-TV returned higher AUC-RIP
scores, indicating that they were easier to reconstruct than the random or
fully open patterns. LwC-RIP and LwC-Inv are the top two methods, with
significantly smaller AUC-RIP values than the random and fully open patterns.
With respect to the trade-off between invertibility and recognition accuracy,
our proposed LwC-RIP outperforms all methods considered.

Figure 6: Subjective verification performances $\downarrow$ at various (image,
pattern) resolutions. The identification accuracies $\uparrow$ were taken from
Table I for references.

###  V-D Subjective Evaluation on Simulated Data

####  V-D1 Experimental Setup

In this perceptual test, we pre-selected a dataset of frontal faces by
retaining the first 1200 identities from the CASIA dataset. For each identity,
we selected the three frontal face images with the highest quality. We
generated corresponding simulated lensless images for face verification by
convolving them with the patterns detailed in Section V-C. We adopted three
configurations for image and pattern size: $(n,m)\in\\{(63\times 63,32\times
32),(127\times 127,32\times 32),(127\times 127,64\times 64)\\}$. We showed a
pair of facial images, with the same identity or different identities, to
human observers, and asked them to judge whether the two images featured the
same subject. We recruited ten observers without prior knowledge of the
identity of the test faces. The ratio between identical and different identity
pairs was 50:50. We used VQOne [46] for the perceptual tests. Each volunteer
labeled a total of 560 pairs of facial images (180 pairs for each size
configuration, which is equivalent to 30 pairs for each coded pattern). Pairs
of facial images for the same coded patterns were shown randomly, to avoid the
possibility that observers may have explicit knowledge about which $H$ pattern
was used. In terms of test condition, the test duration was between 30 and 60
minutes, and matching accuracy was used for verification results.

TABLE II: Top-1 Accuracy (%) with real measurement from selected CASIA dataset.  Imaging |  ResNet18 $\uparrow$ |  Blurriness* $\uparrow$
---|---|---
Measurement size |  64$\times$64 |  512$\times$512 |  64$\times$64 |  512$\times$512
Pinhole | 89.17 | 89.33 | 0.44 | 0.14
Full-open | 46.33 | 45.67 | 0.50 | 0.19
Rand | 55.17 | 54.14 | 0.77 | 0.32
LwoC | 88.00 | 87.67 | 0.82 | 0.36
LwC-TV | 87.00 | 85.50 | 0.80 | 0.40
LwC-RIP | 82.50 | 80.33 | 0.80 | 0.42
*Contrast enhancement impacts blurriness, especially for full-open.

####  V-D2 Subjective Results for Face Verification

Fig. 6 shows that the subjective verification accuracy for pinhole imaging was
highest, while accuracy values for random and fully open patterns were lowest
because details were blurred in the associated images. Verification accuracy
values for LwC-TV and LwC-RIP were similar to the values obtained for random
and fully open patterns. In addition, accuracy values for the learned patterns
increased when human-imperceptible constraints were removed (compare LwoC with
LwC-TV and Lwc-RIP) .

The verification results demonstrate that privacy protection constraints (LwC-
TV and LwC-RIP) could protect privacy as effectively as random or fully open
patterns. Their machine vision accuracy values for identification were also
similar to those associated with pinhole imaging. These experiments confirm
that our proposed losses achieved both visual privacy and high recognition
accuracy. On the other hand, lower image resolution and aperture resolution
ratios lead to improving privacy protection.

###  V-E Identification Results on Real Imaging Systems

We selected the smallest resolution setting (i.e., $\\{63\times 63,32\times
32\\}$) from Table I, and implemented a prototype system with printed patterns
at $d_{1}$ = $22$mm. A plasma monitor for displaying images was placed at a
distance of $80$cm from the printed patterns in a dark room. The coded
patterns were printed with chromium on glass using photolithography printing.
The area of the mask measured $5.6\times 5.6$ mm. Fig. 7 shows the system
prototype and the experimental setup. We chose the ten CASIA classes with the
most images and included the largest images (in terms of pixel size) from each
class in our dataset. The captured images were normalized adaptively to
increase their contrast, cropped to the same size as the mask, and resized to
measure $b\times b$ pixels. We experimented with $b\times b$ = 512$\times$512
and 64$\times$64 pixels. For each identity, 310 and 60 images were captured
for training and testing, respectively, as shown in Fig. 8.

Figure 7:  a) Diagram of the experimental setup, b) the sensor alongside a
photolithography mask on a 5D kinematic mount and c) the optics system in
front of a plasma display in a dark room.  Figure 8: Realcaptured measurements
with various printed patterns: pinhole, full open, learned without constraint
(LwoC), learned with TV loss (LwC-TV), and learned with RIP loss (LwC-RIP).

In real imaging scenarios, image quality is generally poorer than the quality
associated with simulated experiments. Nevertheless, images captured at low
resolution with the pinhole pattern are still relatively sharp, as shown in
figure 8. For other coded patterns, the amount of residual visual information
is in line with the simulations and almost invisible to human vision,
particularly for fully open and random patterns. These results demonstrate
that lensless images successfully prevent human vision from accessing
information about identity. In line with the simulations, we did not record
any substantial privacy information from measurements with fully open and
random patterns (at 50% aperture ratio). Without constraints, the learned
pattern without constraint (LwoC) revealed more information. For the face
recognition application, because of the inevitable difference between real and
simulated measurements, we re-trained a ResNet18 network to handle noise and
low-contrast images. Table II reports the resulting of top-1 accuracy values
achieved on real test data.

Despite its high performance in simulations, pinhole imaging performed poorly
with a real dataset because of inefficient light capture at small coded
ratios. In addition, the captured images had very low contrast because they
were highly blurred. Notwithstanding these shortcomings, pinhole images still
achieved the highest recognition accuracy (89.17%). However, similarly to the
simulated results, subject identity could be retrieved from these images by
the human visual system. In contrast, fully open imaging provided no visual
information but produced the worst performance in the recognition task, with
only 46.33% accuracy. Random patterns yielded slightly better performance at
55.17%, but these values are still substantially smaller (by approximately
28%–33%) than corresponding values obtained with learned patterns without
constraints (LwoC) and learned pattern with constraint (i.e. TV loss/LwC-TV,
RIP loss/LwC-RIP). LwC-TV performed better than LwC-RIP with accuracy values
of 87.00% and 82.50%, respectively, but it also reveals more information as
shown in Fig. 8. In general, performance on real data was consistent with the
results of our simulations, except for an overall reduction in accuracy of
1.2%–5%. The smaller measurement size (64$\times$64) produced more blurriness
and less noise than the larger measurement size (512$\times$512). As a result,
we observed a small reduction in accuracy for Pinhole and a small increase in
accuracy for other patterns.

Although the performance on real data may need to be further improved for
practical applications, this current work has taken a first step in the
direction of developing effective privacy-preserving imaging. Furthermore, we
have successfully implemented a proof-of-concept for our approach in the form
of a physical prototype. In future work, recognition accuracy may be improved
by (1) adopting a larger dataset, (2) developing a better hardware
implementation, and (3) trading visual privacy for higher accuracy.

##  VI Conclusion

This paper proposes a lensless imaging system to protect visual privacy
against perceptual attacks by the human visual system. We achieved this goal
via measurements with similarity loss and or patterns with TV, invertibility,
and RIP losses. Experimental results for both simulation and the hardware
prototype and subjective study have demonstrated that the proposed methods are
able to balance between human-imperceptibility and recognition accuracy for
lensless imaging. We believe this is the first work to construct a learnable
hardware-based recognition system that considers visual privacy protection.
This paper provides preliminary results that outline challenges in combining
strong privacy protection with high recognition accuracy.

## References

  * [1] Padilla-López, J.R., Chaaraoui, A.A., Flórez-Revuelta, F.:  Visual privacy protection methods: A survey.  Expert Systems with Applications 42 (2015) 4177–4195
  * [2] Cavailaro, A.:  Privacy in video surveillance [in the spotlight].  IEEE Signal Processing Magazine 24 (2007) 168–166
  * [3] Wang, S., Cheung, S.c.S., Luo, Y.:  Wearable privacy protection with visual bubble.  In: 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). (2016) 1–6
  * [4] :  Montreal declaration responsive ai_ (2018) Universite de Montreal.
  * [5] Rajpoot, Q.M., Jensen, C.D.:  Security and privacy in video surveillance: Requirements and challenges.  In Cuppens-Boulahia, N., Cuppens, F., Jajodia, S., Abou El Kalam, A., Sans, T., eds.: ICT Systems Security and Privacy Protection, Berlin, Heidelberg, Springer Berlin Heidelberg (2014) 169–184
  * [6] Braun, H., Turaga, P., Spanias, A., Katoch, S., Jayasuriya, S., Tepedelenlioglu, C.:  Reconstruction-Free Compressive Vision for Surveillance Applications.  Morgan & Claypool (2019)
  * [7] Kwan, C., Gribben, D., Tran, T.:  Tracking and classification of multiple human objects directly in compressive measurement domain for low quality optical videos.  In: 2019 IEEE 10th Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON). (2019) 0488–0494
  * [8] Kurakin, A., Goodfellow, I., Bengio, S.:  Adversarial machine learning at scale.  arXiv preprint arXiv:1611.01236 (2016)
  * [9] Yuan, X., He, P., Zhu, Q., Li, X.:  Adversarial examples: Attacks and defenses for deep learning.  IEEE transactions on neural networks and learning systems 30 (2019) 2805–2824
  * [10] Wiyatno, R.R., Xu, A., Dia, O., de Berker, A.:  Adversarial examples in modern machine learning: A review.  arXiv preprint arXiv:1911.05268 (2019)
  * [11] Winkler, T., Erdelyi, A., Rinner, B.:  Trusteye.m4: Protecting the sensor — not the camera.  In: 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). (2014) 159–164
  * [12] Shu, J., Zheng, R., Hui, P.:  Cardea: Context-aware visual privacy protection for photo taking and sharing.  In: Proceedings of the 9th ACM Multimedia Systems Conference. MMSys ’18, New York, NY, USA, Association for Computing Machinery (2018) 304–315
  * [13] Pittaluga, F., Koppal, S.J.:  Privacy preserving optics for miniature vision sensors.  In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2015) 314–324
  * [14] Pittaluga, F., Zivkovic, A., Koppal, S.J.:  Sensor-level privacy for thermal cameras.  In: 2016 IEEE International Conference on Computational Photography (ICCP). (2016) 1–12
  * [15] Huang, G., Jiang, H., Matthews, K., Wilford, P.:  Lensless imaging by compressive sensing.  In: 2013 IEEE International Conference on Image Processing. (2013) 2101–2105
  * [16] Petrovici, M., Damian, C., Udrea, C., Garoi, F., Coltuc, D.:  Single pixel camera with compressive sensing by non-uniform sampling.  In: 2016 International Conference on Communications (COMM). (2016) 443–448
  * [17] Asif, M.S., Ayremlou, A., Sankaranarayanan, A., Veeraraghavan, A., Baraniuk, R.G.:  Flatcam: Thin, lensless cameras using coded aperture and computation.  IEEE Transactions on Computational Imaging 3 (2017) 384–397
  * [18] Boominathan, V., Adams, J.K., Asif, M.S., Avants, B.W., Robinson, J.T., Baraniuk, R.G., Sankaranarayanan, A.C., Veeraraghavan, A.:  Lensless imaging: A computational renaissance.  IEEE Signal Processing Magazine 33 (2016) 23–35
  * [19] Tan, J., Khan, S.S., Boominathan, V., Byrne, J., Baraniuk, R., Mitra, K., Veeraraghavan, A.:  Canopic: Pre-digital privacy-enhancing encodings for computer vision.  In: 2020 IEEE International Conference on Multimedia and Expo (ICME). (2020) 1–6
  * [20] Cambareri, V., Mangia, M., Pareschi, F., Rovatti, R., Setti, G.:  Low-complexity multiclass encryption by compressed sensing.  IEEE Transactions on Signal Processing 63 (2015) 2183–2195
  * [21] Yu, J., Zhang, B., Kuang, Z., Lin, D., Fan, J.:  iprivacy: Image privacy protection by identifying sensitive objects via deep multi-task learning.  IEEE Transactions on Information Forensics and Security 12 (2017) 1005–1016
  * [22] Koelle, M., Ananthanarayan, S., Czupalla, S., Heuten, W., Boll, S.:  Your smart glasses’ camera bothers me! exploring opt-in and opt-out gestures for privacy mediation.  In: Proceedings of the 10th Nordic Conference on Human-Computer Interaction. NordiCHI ’18, New York, NY, USA, Association for Computing Machinery (2018) 473–481
  * [23] Calderbank, R., Jafarpour, S., Schapire, R.:  Compressed learning: Universal sparse dimensionality reduction and learning in the measurement domain.  preprint (2009)
  * [24] Lohit, S., Kulkarni, K., Turaga, P.:  Direct inference on compressive measurements using convolutional neural networks.  In: 2016 IEEE International Conference on Image Processing (ICIP). (2016) 1913–1917
  * [25] Huang, L., Kulkarni, K., Jha, A., Lohit, S., Jayasuriya, S., Turaga, P.K.:  CS-VQA: visual question answering with compressively sensed images.  In: 2018 IEEE International Conference on Image Processing (ICIP). (2018)
  * [26] Lohit, S., Kulkarni, K., Turaga, P., Wang, J., Sankaranarayanan, A.C.:  Reconstruction-free inference on compressive measurements.  In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). (2015) 16–24
  * [27] Zhang, Y., Lu, Y., Nagahara, H., Taniguchi, R.i.:  Anonymous camera for privacy protection.  In: 2014 22nd International Conference on Pattern Recognition, IEEE (2014) 4170–4175
  * [28] Canh, T.N., Jeon, B.:  Privacy-preserving compressive sensing for still images.  In: IEIE International Technical Conference on Circuit/System, Computer, and Communication (ICT-CSCC). (2017) 1–4
  * [29] Tan, J., Niu, L., Adams, J.K., Boominathan, V., Robinson, J.T., Baraniuk, R.G.:  Face detection and verification using lensless cameras.  IEEE Transaction on Computational Imaging (TCI) 5 (2019) 180–194
  * [30] Canh, T.N., Nagahara, H.:  Deep compressive sensing for visual privacy protection in flatcam imaging.  In: The IEEE International Conference on Computer Vision (ICCV) Workshops. (2019)
  * [31] Wang, Z.W., Vineet, V., Pittaluga, F., Sinha, S.N., Cossairt, O., Bing Kang, S.:  Privacy-preserving action recognition using coded aperture videos.  In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. (2019)
  * [32] Luo, Z., Zhou, W., Shi, L., Chen, Z.:  No-reference light field image quality assessment based on micro-lens image.  In: 2019 Picture Coding Symposium (PCS). (2019) 1–5
  * [33] Duong, V.V., Canh, T.N., Jeon, B.:  Light field image coding for efficient refocusing.  In: 2018 IEEE 12th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC). (2018) 74–78
  * [34] Lam, E.Y., Goodman, J.W.:  Iterative statistical approach to blind image deconvolution.  J. Opt. Soc. Am. A 17 (2000) 1177–1184
  * [35] Candès, E.J., Romberg, J.K., Tao, T.:  Stable signal recovery from incomplete and inaccurate measurements.  Communications on Pure and Applied Mathematics 59 (2006) 1207–1223
  * [36] Canh, T.N., Jeon, B.:  Restricted structural random matrix for compressive sensing.  Signal Processing: Image Communication 90 (2021) 116017
  * [37] Crete, F., Dolmiere, T., Ladret, P., Nicolas, M.:  The blur effect: perception and estimation with a new no-reference perceptual blur metric.  In: Human vision and electronic imaging XII. Volume 6492., International Society for Optics and Photonics (2007) 64920I
  * [38] Iliadis, M., Spinoulas, L., Katsaggelos, A.K.:  Deepbinarymask: Learning a binary mask for video compressive sensing.  Digital Signal Processing 96 (2020) 102591
  * [39] Okawara, T., Yoshida, M., Nagahara, H., Yagi, Y.:  Action recognition from a single coded image.  In: 2020 IEEE International Conference on Computational Photography (ICCP). (2020) 1–11
  * [40] Canh, T.N., Khanh Quoc, D., Jeon, B.:  Total variation reconstruction for kronecker compressive sensing with a new regularization.  In: 2013 Picture Coding Symposium (PCS). (2013) 261–264
  * [41] Canh, T.N., Jeon, B.:  Multi-scale deep compressive imaging.  IEEE Transactions on Computational Imaging 7 (2021) 86–97
  * [42] Yang, A., Sankaranarayanan, A.C.:  Designing display pixel layouts for under-panel cameras.  IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (2021) 2245–2256
  * [43] Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.:  Vggface2: A dataset for recognising faces across pose and age.  In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), IEEE (2018) 67–74
  * [44] Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.:  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In: European conference on computer vision, Springer (2016) 87–102
  * [45] Yi, D., Lei, Z., Liao, S., Li, S.Z.:  Learning face representation from scratch.  arXiv preprint arXiv:1411.7923 (2014)
  * [46] Nuutinen, M., Virtanen, T., Rummukainen, O., Häkkinen, J.:  Vqone matlab toolbox: A graphical experiment builder for image and video quality evaluations : Vqone matlab toolbox.  Behavior Research Methods 48 (2015)