# Deep Joint Encryption and Source-Channel Coding: An Image Privacy Protection
Approach

Jialong Xu, Bo Ai, Wei Chen, Ning Wang, and Miguel Rodrigues (corresponding
authors: Bo Ai; Wei Chen)Jialong Xu is with the State Key Laboratory of Rail
Traffic Control and Safety, Beijing Jiaotong University, Beijing 100044, China
(e-mail: jialongxu@bjtu.edu.cn).Bo Ai is with the State Key Laboratory of Rail
Traffic Control and Safety, Beijing Jiaotong University, Beijing 100044, China
(e-mail: boai@bjtu.edu.cn).Wei Chen is with the State Key Laboratory of Rail
Traffic Control and Safety, Beijing Jiaotong University, Beijing 100044, China
(e-mail: weich@bjtu.edu.cn).Ning Wang is with School of Information
Engineering, Zhengzhou University, Zhengzhou 450001,China (e-mail:
ienwang@zzu.edu.cn).Miguel Rodrigues is with the Department of Electronic and
Electrical Engineering, University College London, London, WC1E 7JE, U.K.
(e-mail: m.rodrigues@ucl.ac.uk).

###### Abstract

Joint source and channel coding (JSCC) has achieved great success due to the
introduction of deep learning. Compared with traditional separate source
channel coding (SSCC) schemes, the advantages of DL based JSCC (DJSCC) include
high spectrum efficiency, high reconstruction quality, and the relief of
“cliff effect”. However, it is difficult to couple encryption-decryption
mechanisms with DJSCC in contrast with traditional SSCC schemes, which hinders
the practical usage of the emerging technology. To this end, our paper
proposes a novel method called DL based joint encryption and source-channel
coding (DJESCC) for images that can successfully protect the visual
information of the plain image without significantly sacrificing image
reconstruction performance. The idea of the design is using a neural network
to conduct image encryption, which converts the plain image to a visually
protected one with the consideration of its interaction with DJSCC. During the
training stage, the proposed DJESCC method learns: 1) deep neural networks for
image encryption and image decryption, and 2) an effective DJSCC network for
image transmission in encrypted domain. Compared with the perceptual image
encryption methods with DJSCC transmission, the DJESCC method achieves much
better reconstruction performance and is more robust to ciphertext-only
attacks.

###### Index Terms:

Image encryption, image transformation, privacy protection, security, joint
source-channel coding, deep learning.

##  I Introduction

The modular design principle based on Shannon’s separation theorem [1] is the
cornerstone of modern communications and has enjoyed great success in the
development of wireless communications. However, the assumptions of unlimited
codeword length, delay and complexity in the separation theorem are not
possible in real wireless environments, leading to sub-optimal separate source
channel coding (SSCC). Moreover, for time varying channels, when the channel
quality is lower than the target channel quality, the SSCC cannot decode any
information due to the collapse of channel coding; When the channel quality is
higher than the target quality, the separation coding cannot further improve
the reconstruction quality. This is the famous “cliff effect” [2], which
increases the cost of SSCC during the wireless transmission. In the past
years, joint source-channel coding (JSCC) has been demonstrated theoretically
to have better error exponent than SSCC in discrete memoryless source channels
[3, 4, 5, 6] motivating the development of various JSCC designs over the years
[7, 8, 9, 10] are proposed.

More recently, deep learning (DL) based approaches have been proposed for
source coding[11, 12, 13, 14], channel coding [15, 16, 17, 18], and JSCC [19,
20, 21, 22]. Compared with the SSCC scheme (e.g., JPEG/JPEG2000 for image
coding and LDPC for channel coding), the DL based JSCC (DJSCC) scheme designed
in [19] has better image restoration quality especially in the low signal-to-
noise ratio (SNR) regime. To well adapt variable bandwidth and exploit the
utility of channel output feedback in real wireless environments, the schemes
of DJSCC with adaptive-bandwidth image transmission and image transmission
with channel output feedback are proposed by [20] and [21], respectively.
However, all the aforementioned schemes are trained and inferred at the same
channel conditions (the single SNR) to ensure optimality, demanding the use of
multiple trained networks to suit a range of SNR that leads to considerable
storage requirements in transceivers. To overcome this challenging problem in
DJSCC, [22] proposed a single network for DJSCC which can adapt to a wide
range of SNR conditions to meet the memory limit of the device in real
wireless scenarios. So far, by using the data-driven approach, DJSCC
successfully reduces the difficulty of coding design in traditional JSCC, and
balances the performance and the storage requirement. These benefits brought
by DL make DJSCC methods easier to be employed and deployed than traditional
JSCC schemes.

Figure 1: The DJSCC based wireless communication system.

Yet, to protect information privacy and confidentiality, one must also couple
encryption and decryption mechanisms with DJSCC based wireless communication
systems as illustrated in Fig. 1. The image owner intends to transmit a plain
image (an unencrypted image) to the image recipient through the network
containing a untrusted wired network and a wireless transmission service. To
protect visual information of the plain image, the image owner encrypts the
plain image into an encrypted image (the visually protected image) before
providing it to the wireless service provider. Then the encrypted image is
transmitted by the wireless service provider through the DJSCC transmission.
After the DJSCC wireless transmission, the corrupted encrypted image (the
decoded image by DJSCC decoder) is transmitted to the image recipient through
the untrusted network. The image recipient decrypts the corrupted encrypted
image to the plain image. Even if the encrypted image or the decoded image is
leaked or stolen during the wired network transmission process, visual
information of the plain image can not be acquired directly.

Figure 2: Different Strategies for protecting image privacy in SSCC and DJSCC.
(a) Encrypt the data encoded by source encoder before the channel encoder in
SSCC;(b) Encrypt the image source before the source encoder in SSCC; (c)
Encrypt the image source before the source encoder in DJSCC.

In SSCC, there are two strategies to safeguard image privacy: 1) encrypting
the data encoded by source encoder before the channel encoder as shown in Fig.
2, and 2) encrypting the image source before the source encoder as shown in
Fig. 2 [23]. The first strategy regarded as compression-then-encryption (CtE)
applies the source encoder to compress the image source to the binary data,
and then uses some encryption method (e.g., data encryption standard [24],
advanced encryption standard [25], and Rivest–Shamir–Adleman [26]) to encrypt
the binary data generating the ciphertext. The second strategy called
encryption-then-compression (EtC) is fit for the typical scenario, where the
image provider only takes care of protecting the image privacy and the
telecommunications provider has an overriding interest on improving spectrum
efficiency. Following the EtC strategy, various perceptual image encryption
methods have been developed via transforming an original image to a visually
protected image [27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,
42, 43, 44, 45, 46, 23, 47, 48].

To protect image privacy for DJSCC transmission, the encryption module should
be placed in front of deep joint source-channel encoder as shown in Fig. 2,
which is akin to the position of encryption module in Fig. 2. However, a major
issue is that EtC change the visual structure of the plain image, causing
DJSCC transmission degradation. To the best of our knowledge, there are no
works addressing the encryption problem in the DJSCC framework. In [48, 49,
50, 51, 52], different encryption methods are developed for DNN-based
classification tasks. In addition, Homomorphic Encryption (HE) [53] is a
promising privacy preserving computation method. Some of the HE-based methods
have been applied in DL domain [54, 55, 56, 57]. However, the high
computational complexity, the loss of calculation accuracy due to the use of
polynomial instead of nonlinear activation function, and the large ciphertext
expansion when combining HE with DNNs is an obstacle to the use of HE in
DJSCC.

In this paper, we design a DL based joint encryption and source-channel coding
method that can generate a visually protected image suitable for DJSCC
transmission with high reconstruction performance. The inspiration of our
proposed method originates from image transformation illustrated in Fig. 3.
Image transformation has been successfully applied in image compression (e.g.,
JPEG [58], JPEG2000 [59]) and image processing (e.g., image classification,
image semantic segmentation [60]) for facilitating the subsequent operations.
Although the transformed image contains the information of the plain image,
little visual information can be perceived due to the transformation. Compared
with the existing perceptual image encryption methods, the proposed method not
only protects the image privacy, but also have a better reconstruction
performance. Another advantage of our proposed method is that the proposed
method is more robust against the ciphertext-only attacks than the existing
perceptual image encryption methods. Moreover, the full convolutional
architecture of the proposed method makes it more flexible to deal with images
of different sizes without loss of the visual protection performance.

Figure 3: Illustration of image transformation. (a) The plain image, (b) The
transformed image using discrete cosine transform (DCT).

The rest of this paper is organized as follows. Section II presents related
work on deep joint source-channel coding, perceptual image encryption, and
image transformation. Then, the proposed method is presented in Section III.
In Section IV, the proposed method is evaluated on datasets with low
resolution and high resolution, respectively. Visual security of the proposed
is evaluated by ciphertext only attacks in Section V. Finally, Section VI
concludes this paper.

##  II Related work

###  II-A Deep Joint Source Channel Coding

As shown in the lower part of Fig. 1, DJSCC follows the end-to-end autoencoder
architecture [61] which replaces the source encoder and the channel encoder
(with/without the modulation) with a deep joint source-channel encoder (DJSCE)
in the transmitter, and similarly, replaces the corresponding source decoder,
the channel decoder (with/without demodulation) by a deep joint source-channel
decoder (DJSCD).

The initial DJSCC work proposed a recurrent neural network (RNN) based DJSCE
and DJSCD for text transmission over binary erasure channels [62]. From then
on, DJSCC attracted increasing interests, especially for image compression and
transmission. In [19], fully convolutional neural networks (FCNNs) are used
for DJSCE and DJSCD, and are shown to outperform SSCC method especially in the
low SNR regime. Furthermore, the DJSCC method in [19] provides a graceful
performance degradation in communications scenarios suffering from large
channel estimation errors, associated with the well-known “cliff effect” in
SSCC.

In the wireless communication systems, the transmission bandwidth is always
dynamically allocated according to user requirements and system overload.
DJSCC-l proposed by [20] can transmit progressively in layers; When the
available bandwidth is limited, codewords of the first few layers are
transmitted to the receiver to reconstruct the transmitted image with lower
quality; When the available bandwidth is increased, codewords of the residual
layers are transmitted to the receiver and are combined with the first
codewords to reconstruct the transmitted image with higher quality.

Once channel output feedback is available, DJSCC-f proposed by [21] can
further improve the reconstruction quality by exploiting the channel output
feed back. The difference between DJSCC-l and DJSCC-f is that DJSCC-f not only
aggregates the layered transmission information at the receiver, but also
jointly processes the transmitted image and the processed channel output
feedback at the transmitter. Compared with DJSCC without channel output
feedback, DJSCC with both noisy and noiseless channel output feedback can
achieve considerable performance gains. However, [19, 21, 20] must train
multiple networks in a range of SNR and select one of them depending on the
real SNR condition to keep their optimality, which would lead to heavy burden
on the storage overhead of devices.

Inspired by the resource assignment strategy in traditional JSCC, [22]
introduces the SNR feedback to DJSCC and proposes a general DJSCC method named
Attention DL based JSCC (ADJSCC). ADJSCC can dramatically reduce the storage
overhead while maintaining the similar performance by using attention
mechanisms. In addition, [63] proposed a DJSCC based on maxmizing the mutual
information between the source and noisy received codeword for the binary
erasure channel and the binary symmetric channel. [64] and [65] model their
DJSCC systems via variational autoencoder and manifold variational
autoencoders for a Gaussian source, respectively. However, all of the
aforementioned work are not designed to guarantee information privacy
protection.

###  II-B Perceptual Image Encryption

Perceptual image encryption aims to transform a plain image into an encrypted
image, which downgrades visual quality to protect the original perceptual
information.

One can change image visual information along two dimensions: the space
dimension and the pixel dimension. The space based encryption involves re-
arranging (scrambling) the different pixels within the image. There are
numerous image scrambling algorithms, e.g., Arnold [27], Baker Transformation
[28], Fibonacci Transformation [29], Magic Square [30], RGB Scramble [31],
Chaos Scramble [32], and SCAN Pattern [33]. The pixel based encryption only
changes pixel values to protect visual information. Popular methods include
DES[34], Hill algorithm [35], chaotic logistic map[36], and cellular automata
[37] based methods. Combining space based encryption and pixel based
encryption can make encryption more robust. Well-known techniques include
Arnold and Chen’s chaotic system based method [38], compound chaotic sequence
based method [39], and 3D chaotic map based method [40]. However, the
aforementioned encryption methods do not take into account the effect of
subsequent operations such as source or channel coding.

Considering the compression task in EtC system, stream cipher methods using a
pseudo-random key generator followed by Slepian-Wolf coding and resolution
progressive compression have been used for lossless compression in [42, 43].
To further improve the compression ratio, pixel based image encryption
followed by a lossy scalable compression technique [44], image encryption via
prediction error clustering and random permutation followed by a context-
adaptive arithmetic coding [45], pseudorandom permutation based image
encryption followed by orthogonal transformation based lossy compression [46],
and block scrambling based image encryption followed by JPEG lossy compression
[23] are proposed for lossy compression. However, when the final signal
processing task is changed from image reconstruction to e.g. image
classification, the methods designed for EtC systems stop working.

DL has led to state-of-the-art performance in various image processing tasks,
motivating more recently the application of deep learning techniques to
encrypted images. The methods proposed by [48, 49, 50, 51, 52] are designed
for DL based classification. Tanaka’s method [48] is a hybrid encryption
method, adding an adaptation network prior to DNNs. The pixel based image
encryption methods with/without key management [49, 50] can directly be fed
into DNNs for classification. DL based encryption methods, e.g., generative
adversarial networks (GAN) and DNNs, are proposed for image encryption in [51]
and [52], respectively. Both of [51] and [52] employ image
transformation—transforms the image from one domain to the other domain—to
protect visual information. However, The image encryption methods designed for
DL based classification are still not fit for DJSCC. The encryption images for
classification only reserve some specific semantic information relevant to
image class while the pixel based information of the image is discarded, which
cause the performance degradation for the image reconstruction task, i.e.,
DJSCC.

##  III Deep Joint Encryption and Source-Channel Coding

Our method motivation is to successfully protect the visual information of the
plain image without significantly sacrificing image reconstruction performance
in DJSCC. Conventional image transformations, e.g., discrete cosine transform
(DCT) and discrete wavelet transform (DWT), can provide the convenience for
subsequent operations and simultaneously generate a visually protected image
in the transformation. However, [66, 67, 60] demonstrate that conventional
image transformations are hardly matched with the DNNs if the structures of
DNNs are not modified. In this Section, a DL based joint encryption and
source-channel coding (DJESCC) method is proposed for protecting visual
information of plain images in DJSCC transmission.

###  III-A System Model

Figure 4: The system model of the proposed DL based joint encryption and
source-channel coding method.

Considering a visually protected DJSCC transmission system as shown in the
lower part of Fig. 4, a plain image is represented by
$\boldsymbol{x}\in\mathbb{R}^{n}$, where $\mathbb{R}$ denotes the set of real
numbers and $n=h\times w\times c$. $h,w,c$ denote the width, height, and the
number of channels of an image, respectively. The encryption network
transforms the plain image into a visually protected image. This encryption
process can be expressed as:

$\boldsymbol{y}=e_{\boldsymbol{\mu}}(\boldsymbol{x})\in\mathbb{R}^{n},$ (1)

where $e_{\boldsymbol{\mu}}(\cdot)$ represents an encryption deep neural
network based parameterized by the set of parameters $\boldsymbol{\mu}$. Note
that the plain image $\boldsymbol{x}$ and the visually protected image
$\boldsymbol{y}$ have the same size.

Then the visually protected image $\boldsymbol{y}$ is encoded by the joint
source-channel encoder as:

$\boldsymbol{z}=f_{\boldsymbol{\theta}}(\boldsymbol{y})\in\mathbb{C}^{k},$ (2)

where $\mathbb{C}$ denotes the set of complex numbers, $k$ represents the size
of channel input symbols, and $f_{\boldsymbol{\theta}}(\cdot)$ represents an
joint source-channel encoder parameterized by the set of parameters
$\boldsymbol{\theta}$. The encoded complex-valued $\boldsymbol{z}$ represents
the transmitted signals at the transmitter. The real parts and the imaginary
parts of $\boldsymbol{z}$ are considered as in-phase components I and
quadrature components Q of the transmitted signals, respectively. Due to the
average power constraint at the transmitter,
$\frac{1}{k}\mathbb{E}(\boldsymbol{zz^{*}})\leq 1$ must be satisfied, where
$\boldsymbol{z^{*}}$ is the complex conjugate transpose of $\boldsymbol{z}$.

The transmitted signals are corrupted by the wireless channel. We adopt a well
known AWGN model111By applying equalization at the receiver, the flat fading
channel model can be represented as AWGN model, while the noise has a
different distribution. given by:

$\boldsymbol{\hat{z}}=\eta(\boldsymbol{z})=\boldsymbol{z}+\boldsymbol{\omega},$
(3)

where $\boldsymbol{\hat{z}}\in\mathbb{C}^{k}$ is the channel output and
$\boldsymbol{\omega}\in\mathbb{C}^{k}$ denotes the additive noise modeled by
$\boldsymbol{\omega}\sim\mathbb{CN}(0,\sigma^{2}\boldsymbol{I})$ where
$\sigma^{2}$ represents the average noise power and $\mathbb{CN}(\cdot,\cdot)$
denotes a circularly symmetric complex Gaussian distribution.

In turn, the channel output symbols $\boldsymbol{\hat{z}}$ is decoded by the
joint source-channel decoder as:

$\boldsymbol{\hat{y}}=g_{\boldsymbol{\phi}}(\boldsymbol{\hat{z}})\in\mathbb{R}^{n},$
(4)

where $g_{\boldsymbol{\phi}}(\cdot)$ represents an joint source-channel
decoder parameterized by the set of parameters $\boldsymbol{\phi}$. The
decoded image $\boldsymbol{\hat{y}}\in\mathbb{R}^{n}$ is with the same size of
the visually protected image $\boldsymbol{y}$.

Similarly to the encryption step, the decryption network is employed to
convert the decoded image to the decrypted image, as follows:

$\boldsymbol{\hat{x}}=d_{\boldsymbol{\nu}}(\boldsymbol{\hat{y}})\in\mathbb{R}^{n},$
(5)

where $d_{\boldsymbol{\nu}}(\cdot)$ represents a decryption deep neural
network parameterized by the set of parameters $\boldsymbol{\nu}$ and the
decrypted image $\boldsymbol{\hat{x}}\in\mathbb{R}^{n}$ is an estimation of
the plain image. The bandwidth ratio $R$ is defined as $k/n$, where $n$ is the
source size (i.e., image size) and $k$ is the channel bandwidth (i.e., channel
input size).

###  III-B The Proposed Method

In sharp contrast with DJSCC methods [19, 21, 20, 22], we require our DJESCC
method to address two issues:

1) To hide visual information of a plain image.

2) To extract effective features from the visually protected image for
subsequent DJSCC transmission.

The classical full-reference metric of image similarity is peak signal-to-
noise ratio (PSNR) between the original image and the restored image, which is
defined as:

$\rm PSNR=10log_{10}\frac{MAX^{2}}{MSE}(dB).$ (6)

where MAX is the maximum possible value of the image pixels and MSE is the
abbreviation of mean square error between the original image and the restored
image. Although the prediction of PSNR performance is not always consistent
with quality perception by the human visual system, the simplicity and the
inexpensive computational complexity make it widely used in the field of image
processing [68].

However, as is illustrated in Fig. 5, PSNR is not a good metric to assess the
effect of hiding visual information. Fig. 5(b) has a lower PSNR than Fig.
5(c), while the visual information (e.g., the birds and the leaf) are more
easily identified in Fig. 5(b) than in Fig. 5(c). To solve this problem, a
feature extraction network is employed to measure the effect of hiding visual
information. The feature extraction method was initially used to measure the
similarity between two images [69] and then successfully used to measure the
difference between two images [52].

Figure 5: PSNR comparison. (a) plain image, (b) invert image (the intensity
values of the plain image are substracted by 255), (c) shuffle image (the
intensity values of the plain image are randomly shuffled in space and channel
dimension).

In the training stage, features of the plain image $\boldsymbol{x}$, the
encrypted image $\boldsymbol{y}$, and the decoded image $\boldsymbol{\hat{y}}$
are extracted by the feature extraction network $h_{\boldsymbol{\psi}}$ in
Fig. 4. The feature loss $\mathcal{L}_{e}$ between the plain image
$\boldsymbol{x}$ and the encrypted image $\boldsymbol{y}$ is expressed as:

$\mathcal{L}_{e}=\frac{1}{m}\lVert
h_{\boldsymbol{\psi}}(\boldsymbol{x})-h_{\boldsymbol{\psi}}(\boldsymbol{y})\rVert_{2}^{2},$
(7)

and the feature loss $\mathcal{L}_{d}$ between the plain image
$\boldsymbol{x}$ and the encrypted image $\boldsymbol{\hat{y}}$ is expressed
as:

$\mathcal{L}_{d}=\frac{1}{m}\lVert
h_{\boldsymbol{\psi}}(\boldsymbol{x})-h_{\boldsymbol{\psi}}(\boldsymbol{\hat{y}})\rVert_{2}^{2},$
(8)

where $h_{\boldsymbol{\psi}}(\boldsymbol{x})\in\mathbb{R}^{m}$,
$h_{\boldsymbol{\psi}}(\boldsymbol{x})\in\mathbb{R}^{m}$, and
$h_{\boldsymbol{\psi}}(\boldsymbol{y})\in\mathbb{R}^{m}$ are the features of
the plain image $\boldsymbol{x}$, the encrypted image $\boldsymbol{y}$, and
the decoded image $\boldsymbol{\hat{y}}$ extracted by the feature extraction
network and $m=h_{f}\times w_{f}\times c_{f}$. $h_{f}$, $w_{f}$, and $c_{f}$
denote the width, height, and the number of channels of an extracted feature,
respectively. The reconstruction loss $\mathcal{L}_{e2e}$ between the plain
image $\boldsymbol{x}$ and the decrypted image $\boldsymbol{\hat{x}}$ is
expressed as:

$\mathcal{L}_{r}=d(\boldsymbol{x},\boldsymbol{\hat{x}})=\frac{1}{n}\lVert\boldsymbol{x}-\boldsymbol{\hat{x}}\rVert_{2}^{2}=\frac{1}{n}\sum_{i=1}^{n}(x_{i}-\hat{x}_{i})^{2},$
(9)

where $x_{i}$ and $\hat{x}_{i}$ represents the intensity values of the plain
image $\boldsymbol{x}$ and the decrypted image $\boldsymbol{\hat{x}}$,
respectively.

Different from the image-to-image translation tasks [51, 69] which minimizes
the feature loss in the training stage, the proposed method maximizes
$\mathcal{L}_{e}$ and $\mathcal{L}_{d}$ to hide visual information for the
encrypted image $\boldsymbol{x}$ and the decoded image $\boldsymbol{\hat{y}}$.
The total loss used to train the proposed method:

$\mathcal{L}_{total}=\mathcal{L}_{r}-\lambda_{e}\mathcal{L}_{e}-\lambda_{d}\mathcal{L}_{d},$
(10)

where $\lambda_{e}\in\mathbb{R}^{+}$ and $\lambda_{d}\in\mathbb{R}^{+}$ are
the weights of $\mathcal{L}_{e}$ and $\mathcal{L}_{d}$, respectively. Under a
certain bandwidth ratio $R$, DJESCC learns the parameters of the encryption
network $\boldsymbol{\mu}$, the deep joint source-channel encoder
$\boldsymbol{\theta}$, the joint source-channel decoder $\boldsymbol{\phi}$,
and the decryption network $\boldsymbol{\nu}$ by minimizing the end-to-end
distortion as follows:

$(\boldsymbol{\mu}^{*},\boldsymbol{\theta}^{*},\boldsymbol{\phi}^{*},\boldsymbol{\nu}^{*})=\mathop{\arg\min}\limits_{\boldsymbol{\mu},\boldsymbol{\theta},\boldsymbol{\phi},\boldsymbol{\nu}}\mathbb{E}_{p(\sigma^{2})}\mathbb{E}_{p(\boldsymbol{x},\boldsymbol{\hat{x}})}(L_{total}),$
(11)

where
$\boldsymbol{\mu}^{*},\boldsymbol{\theta}^{*},\boldsymbol{\phi}^{*},\boldsymbol{\nu}^{*}$
are the optimal parameters, $p(\boldsymbol{x},\boldsymbol{\hat{x}})$
represents the joint probability distribution of the plain image
$\boldsymbol{x}$ and the decrypted image $\boldsymbol{\hat{x}}$, and
$p(\sigma^{2})$ represents the probability distribution of the average noise
power $\sigma^{2}$. Note that the probability distribution of the average
noise power instead the fixed average noise power is adopted due to the
considerations of the storage overhead and the difficulty to acquire the
average noise power in the image owner/recipient. In addition, empirical
average instead of statistic average is adopted in the training stage. During
the training stage, the proposed DJESCC method learns: 1) an effective method
to hide visual information of the plain image, 2) an easy to be extracted
image domain for the subsequent DJSCC transmission, 3) an effective DJSCC
transmission method, and 4) an effective method to reconstruct the plain
image.

After the DJESCC network has been trained, the encryption network
$e_{\boldsymbol{\mu}}$ and the decryption network $e_{\boldsymbol{\mu}}$ are
securely distributed to the image owner and the image recipient by some
security protocol, e.g., Secure Sockets Layer (SSH) protocol, respectively.
The deep joint source-channel encoder and the deep source-channel decoder are
distributed to the DJSCC transmission service provider.

In the test stage, the plain image $\boldsymbol{x}$ is first converted to the
encrypted image $\boldsymbol{y}$ by the image owner using Eq. (1). Then the
encrypted image $\boldsymbol{y}$ is sent to the DJSCC transmission service
provider. DJSCC transmission is executed using Eq. (2) and Eq. (4) and the
decoded image $\boldsymbol{y}$ is obtained. The DJSCC transmission service
provider sends the decoded image $\boldsymbol{\hat{y}}$ to the image
recipient, which uses Eq. (5) to decrypt the decoded image
$\boldsymbol{\hat{y}}$. The process of test stage is illustrated in the lower
part of Fig. 4.

##  IV Experimental Results

Figure 6: The architecture of the DJSCC network[19] adopted in this paper.
Figure 7: The architecture of the encryption and decryption networks adopted
in this paper.

Considering the generality of the DJESCC method, there are multiple
architecture choices for the encryption network, the DJSCC encoder, the DJSCC
decoder, and the decryption network. To prove the potential of our proposed
method, the original DJSCC network architecture in [19] is chosen in the
subsequent experiments. It is worthy noting that our proposed method can be
applied to other extensions of the original DJSCC network architecture.

The DJSCC architecture proposed by [19] is shown in Fig. 6, where the encoder
and decoder networks are adopted in our DJESCC method. The DJSCC encoder
consists of the normalization layer, five alternant convolutional layers and
PReLU layers, the reshape layer, and the power normalization layer. The DJSCC
decoder consists of the reshape layer, five alternanting transposed
convolutional layers and activation layers (i.e., four PReLU layers and one
sigmoid layer), and the denormalization layer. The normalization layer
converts the input image with the pixel value range [0, 255] to the image with
pixel value range [0, 1], and the denormalization layer performs the opposite
operation. The notation $F\times F\times K|S$ in a convolution/transpose
convolution layer denotes that it has $K$ filters with size $F$ and stride
down/up $S$. The power normalization layer is used to satisfy the average
power constraint at the transmitter. The channel number of the last
convolutional layer in the DJSCC encoder is $t$. The bandwidth ratio of the
proposed method is $R=t/96$. The architecture shown in Fig. 7 is employed as
the architectures of the encryption network and the decryption network, which
is a shallow version of U-Net [70]. In the training stage, the parameters of
the feature extraction network is fixed.

Tensorflow [71] and its high-level API Keras is used to implement the proposed
DJESCC method222Source codes for constructing the proposed method are
available at: https://github.com/alexxu1988/DJESCC.. The proposed method is
trained under a uniform distribution within the SNR range [0, 20] dB. The
following experiments run on a Linux server with twelve octa-core Intel(R)
Xeon(R) Silver 4110 CPUs and sixteen GTX 1080Ti GPU. Each experiment was
assigned six CPU cores and a GPU.

We first consider the performance of the proposed method on CIFAR-10 dataset,
which consists of 60000 $32\times 32\times 3$ color images associated with 10
classes where each class has 6000 images. Note the goal of our proposed method
is to generate visually protected images for the untrusted transmission
channels and reconstruct the plain image at the receiver, so the class label
of each image is not used in the following experiments. Training dataset and
test dataset contain 50000 images and 10000 images, respectively. We use a
part of the VGG16 [72] with batch normalization, i.e., a classical network for
classification tasks, as the feature extraction network, which is pretrained
on the CIFAR-10. All of the networks were trained for 500 epochs by using Adam
optimizer with an initial learning rate of $10^{-3}$. Once learning stagnated
for 10 epochs, the learning rate was reduced by a factor of 10. The
performance of the DJESCC networks were evaluated at specific $\rm
SNR_{test}\in$ [0,20] dB on CIFAR-10 test dataset. To alleviate the effect of
the randomness caused by the wireless channel, each image in CIFAR-10 test
dataset is transmitted 10 times. PSNR is used in the evaluation of the
reconstruction performance between the plain image and the decrypted image.

Figure 8: Performance of DJESCC and DJSCC trained on CIFAR-10 training dataset
and evaluated on CIFAR-10 test dataset with R=1/6. Figure 9: Visually
protected images generated by the DJESCC method. The image in the first column
is the plain image. The images in the second column are the encrypted images
transformed by the image owner. The images in the third column are the decoded
images decoded by the DJSCC decoder for $\rm SNR=0,10,20$ dB. The images in
the last column are the decrypted images transformed by the image recipient
for $\rm SNR=0,10,20$ dB. PSNR and structural similarity index (SSIM) are
given under the images. (a) $\lambda_{e}=\lambda_{d}=0$, (b)
$\lambda_{e}=\lambda_{d}=0.005$, (c) $\lambda_{e}=\lambda_{d}=0.05$, (d)
$\lambda_{e}=\lambda_{d}=0.5$.

Fig. 8 compares the reconstruction performance of the proposed method with
different loss weights (e.g., $\lambda_{e}=\lambda_{d}=0.005,0.05,0.5$) at
bandwidth ratio $R=1/6$. The reconstruction performance of the DJSCC without
visual protection, i.e., $\lambda_{e}=\lambda_{d}=0$ is also plotted. The
reconstruction performance is gradually decreased with the increase of
$\lambda_{e}$ and $\lambda_{d}$, as the training promotes protection of visual
information instead of reconstruction of the images. Fig. 9 shows the
visualization of the plain image, the encrypted images transformed by the
image owner, the decoded images decoded by the DJSCC transmission service
provider, and the decrypted images transformed by the image recipient for $\rm
SNR=0,10,20$ dB with different loss weights $\lambda_{e}$ and $\lambda_{d}$.
The plain image comes from CIFAR-10 test dataset. Fig. 9(a) shows that even
though the feature losses between the plain image and the encrypted/decoded
image are not used, the visual information of the plain image is partially
protected in the encrypted image and the decoded image. The outline of the dog
can be vaguely identified in the encrypted image in Fig. 9(a), and less visual
information is captured in the decoded images than in the encrypted image.
With the increase in loss weights $\lambda_{e}$ and $\lambda_{d}$, the visual
protection in the encrypted image is enhanced, and the corresponding visual
pattern in the decoded image is also changed. In Fig. 9(b) with the loss
weights $\lambda_{e}=\lambda_{d}=0.005$, the outline of the dog can be vaguely
identified in the encrypted image. The encrypted image decays to regular black
and white lattice with no visual information of the plain image in Fig. 9(c)
with the loss weights $\lambda_{e}=\lambda_{d}=0.05$. As the loss weights
increase to $\lambda_{e}=\lambda_{d}=0.5$, the left part of the encrypted
image contains alternated red and blue stripes and the right part of it
contains twisted black and white lattice, which is more irregular than the
encrypted image in Fig. 9(d). In Fig. 9(b)(c)(d), all of the decoded images
with different SNRs can protect visual information of the plain image. The
proposed method with $\lambda_{d}=\lambda_{e}=0.05$ shows a better trade-off
between the reconstruction performance and the visual protection performance
than others.

Since our proposed method is the first work that constructs visually protected
image for DJSCC transmission, we compare the proposed method with two
perceptual image encryption methods, i.e. the learnable image encryption (LE)
method [48] and the pixel-based image encryption (PE) method [50], which are
designed for image classification task. The classification accuracy of
ResNet-20 [73] based on the LE method and the PE method are 87.02% and 86.99%
on CIFAR-10 test dataset, respectively [52].

Figure 10: Performance of DJESCC, DJSCC_LE, and DJSCC_PE evaluated on CIFAR-10
test dataset. DJESCC is trained on CIFAR-10 training dataset. Figure 11:
Visually protected images comparison generated by the DJESCC method, the
DJSCC_LE method, and the DJSCC_PE method with R=1/6. The image in the first
column is the plain image. The images in the second column are the encrypted
images transformed by the image owner. The images in the third column are the
decoded images decoded by the DJSCC decoder for $\rm SNR=0,10,20$ dB with
different methods. The images in the last column are the decrypted images
transformed by the image recipient for $\rm SNR=0,10,20$ dB with different
methods. PSNR and SSIM are given under images. (a) DJESCC with
$\lambda_{e}=\lambda_{d}=0.05$, (b) DJSCC_LE, (c) DJSCC_PE.

Fig. 10 compares the DEJSCC method with the LE based DJSCC (DJSCC_LE) method
and the PE based DJSCC (DJSCC_PE) method for bandwidth ratios R= 1/6 and
R=1/12. The performance of the DJSCC_PE with R=1/6 is slightly increased
around 11.5 dB as the SNR increases from 0dB to 20dB, which is much lower than
that of the DJESCC with $\lambda_{e}=\lambda_{d}=0.05$ and R=1/6. Although the
performance of the DJSCC_PE with R=1/6 is better than that of the DJSCC_LE
with R=1/6, the performance of the DJSCC_PE with R=1/6 is still 5 dB lower
than that of the DJESCC at $\rm SNR_{test}=0$ dB and the performance gap
between the DJSCC_PE and the DJESCC with $\lambda_{e}=\lambda_{d}=0.05$ is
further widened with the increase of the SNR. Comparing the methods with
R=1/12 shows the similar results. With the increase of R from 1/12 to 1/6,
DJESCC achieves more gains than DJSCC_PE and DJSCC_LE. Fig. 11 shows the
corresponding visual performance for DJESCC, DJSCC_LE, DJSCC_PE with $\rm
SNR=0,10,20$ dB and R=1/6. Different from the black and white lattice
characteristic of the encrypted image shown in the DJESCC method, the
encrypted images of the DJSCC_LE method and the DJSCC_PE method are shown as
noisy images. However, the pixels in the part of the encrypted images
corresponding to that in the dog part of the plain image show less randomness
than the pixels in other parts. The decoded images of the DJSCC_LE method and
the DJSCC_PE method are similar to the corresponding encrypted images of the
DJSCC_LE method and the DJSCC_PE method, respectively. The decrypted images of
the DJSCC_LE method at $\rm SNR=0,10,20$ dB are disturbed by some noisy
pixels, while the decrypted images of the DJSCC_PE method at $\rm SNR=0,10,20$
look blurred.

##  V Visual Security of The Proposed Method

As described in Section I, it is unsafe during the network transmission. The
visually protected images—the encrypted image generated by the image owner
then sent to the DJSCC provider through the Internet and the decoded image
processed by the DJSCC provider then sent to the image recipient—may be leaked
or stolen in the wired network. Hence the cipher-text-only attack (CoA), where
the attacker has access to decoded image and wishes to recover plain image,
must be considered to evaluate the visual security of the proposed method. In
this paper, we focus on two CoAs: feature reconstruction (FR) attack and
generative adversarial network based (GAN-based) attack.

###  V-A FR Attack

Based on the fact that many visually protected methods deal with the plain
image in patch (e.g., the LE method operates the pixels in a 4$\times$4 block)
or in pixel (e.g., the PE method only deal with the pixels without change
their positions), FR attack proposed by [74] is employed to reconstruct the
edge information of plain images from the encrypted images. The algorithm of
FR attack is described in Algorithm 1, where $\lfloor\cdot\rfloor$ denotes the
floor function.

Algorithm 1 The FR-attack method [74]

0: the 8-bit visually protected image $\boldsymbol{y}\in\mathbb{R}^{h\times
w\times 3}$, leading bit $b\in\\{0,1\\}$

0: the 8-bit restored image $\boldsymbol{\hat{x}}\in\mathbb{R}^{h\times
w\times 3}$

1: the 8-bit RGB image is divided into $m=h\times w$ pixels,
$y_{(i)}\in\mathbb{R}^{1\times 1\times 3}$ represents the $i$-th pixel

2: for i = 0 : 1 : $h\times w$ do

3: for j = 0 : 1 : 2 do

4: if $\lfloor y_{i}[j]\rfloor\neq b$ then

5: $\hat{x}_{i}[j]=y_{i}[j]\oplus(2^{8}-1)$

6: end if

7: end for

8: end for

9: assemble $\hat{x}_{i},i\in[1,m]$ to form
$\boldsymbol{\hat{x}}\in\mathbb{R}^{h\times w\times 3}$

10: return $\boldsymbol{\hat{x}}$

###  V-B GAN-based Attack

GAN-based attack adopted in [75] has shown state-of-the-art attack performance
for CoA. As illustrated in Fig. 12, the Generator intends to generate images
from encrypted images to cheat the discriminator while the purpose of the
discriminator is to well distinguish whether the images fed into it originates
from the plain image dataset or from the decrypted images. In the training
stage, the training dataset containing plain images is divided into
$\boldsymbol{X}_{1}$ and $\boldsymbol{X}_{2}$ with the same size.
$\boldsymbol{X}_{2}$ is encrypted by some encryption method. Both
$\boldsymbol{X}_{1}$ and $E(\boldsymbol{X}_{2})$ are employed for
alternatively training the generator and the discriminator. Note that paired
images are not required in the training stage. With the progress of training,
the generator can gradually generate the images more and more like plain
images from the encrypted images. In other words, the generator decrypts the
encrypted images and gets the visual information.

Figure 12: GAN-based attack.

The generators proposed in [75] are used as generators of GAN-based attack for
the DJSCC_LE method and the DJSCC_PE method except using the sigmoid
activation instead of tanh activation in the last layer. The transformation
architecture shown in Fig. 7 is employed as the generator for the DJESCC
method. The discriminator contains three convolutional layers with kernel size
2, stride 2, the number of channels 16, 16, 32 and leaky ReLU activation, and
a dense layer with sigmoid activation. The models of GAN-based attack networks
were trained for 600 epochs by using the Adam optimizer with the learning rate
0.0001 and batch size with 64.

###  V-C Visual Performance of CoAs

We employ the STL-10 dataset in the expriments for a better visual comparison.
Full convolutional architecture based DJSCC can deal with tasks of different
image sizes in the test phase [19] and [22]. The DJESCC model is trained on
CIFAR-10, which is unknown to the attacker. The LE method and the PE method
are size sensitive. Hence the DJSCC_LE method and the DJSCC_PE method were
trained on the STL-10 dataset under a uniform distribution within the SNR
range [0, 20] dB. FR attack can be directly employed on the cipher images (the
encrypted image and the decoded image generated by DJESCC, DJSCC_LE, or
DJSCC_PE), while the GAN-based attack should first be trained and then be
employed on the cipher images. The GAN-based attack was trained and tested on
STL10 dataset for DJESCC, DJSCC_LE and DJSCC_PE, respectively.

Visual information reconstructed by FR attack and GAN-based attack with $\rm
SNR_{test}$=10 dB are shown in Fig. 13. For attacking encrypted images by FR
attack, edge information against the PE method can be successfully
reconstructed and edge information against the LE method can be vaguely
reconstructed. However, the black image generated by FR attack against the
DJESCC method shows that FR attack can not reconstruct any edge information
from the encrypted image generated by the DJESCC method. Similar results can
be found for attacking decrypted images by FR attack, except that the edge
information can not be reconstructed by FR attack attack against the PE method
because of the low DJSCC transmission quality in the DJSCC_PE method. GAN-
based attack can successfully recover visual information from the encrypted
image protected by the DJSCC_LE method, the decrypted image protected by the
DJSCC_LE method, and the encrypted image protected by the DJSCC_PE method,
while little visual information is recovered from the encrypted image and the
decrypted image protected by the DJESCC method. Compared with the DJSCC_LE
method and the DJSCC_PE method, the visual protected images (i.e.,the
encrypted image and the decoded image) generated by the DJESCC method are more
robust against FR attack and GAN-based attack.

Figure 13: Reconstructed visual information from the encrypted image and the
decoded image by FR attack and GAN-based attack at $\rm SNR_{test}$=10 dB. The
image in the first column is the plain image. The images in the second column
are the encrypted image, FR attack for the encrypted image and GAN-based
attack for the encrypted image. The images in the third column are the decoded
image, the FR-attack for the decode image, and the GAN-attack for the decoded
image. decoded by the DJSCC decoder for SNR= 0, 10, 20 dB. PSNR and structural
similarity index (SSIM) are given under the images. (a) DJESCC with
$\lambda_{e}=\lambda_{d}=0.05$, (b) DJSCC_LE, (c) DJSCC_PE.

##  VI Conclusion

Inspired by image transformation, we have proposed a novel DJESCC method. By
applying end-to-end training, the proposed DJESCC method learned two DNNs to
transform the images, i.e., one transformed from the plain image domain to the
encrypted image domain for encryption, and the other one transformed from the
DJSCC decoded image domain for decryption. Besides that, the proposed DJESCC
method simultaneously learned an effective DJSCC transmission method in the
encryption domain during the training stage.

Compared with perceptual image encryption methods (e.g., the LE method and the
PE method) for DJSCC, the proposed DJESCC method has shown a much better
reconstruction performance. In addition, the experimental results demonstrated
that the proposed DEJSCC method were robust to the two types of CoA, i.e., the
FR attack and the GAN-based attack. It is worth noting that the proposed
DJESCC method is a general method for protecting visual information of the
plain image transmitted via DJSCC. With some appropriate modifications of the
DJESCC, the proposed mechanism can be applied to various different DJSCC
architectures, e.g., the DJSCC-l [20], the DJSCC-f [21], and the ADJSCC [22].

## References

  * [1] T. M. Cover, _Elements of Information Theory_. John Wiley & Sons, 1999.
  * [2] M. Skoglund, N. Phamdo, and F. Alajaji, “Hybrid digital–analog source–channel coding for bandwidth compression/expansion,” _IEEE Transactions on Information Theory_ , vol. 52, no. 8, pp. 3757–3763, 2006.
  * [3] R. G. Gallager, _Information Theory and Reliable Communication_. Springer, 1968, vol. 2.
  * [4] I. Csiszár, “Joint source-channel error exponent,” _Problems of Control & Information Theory_, vol. 9, pp. 315–328, 1980.
  * [5] Y. Zhong, F. Alajaji, and L. L. Campbell, “Joint source–channel coding error exponent for discrete communication systems with markovian memory,” _IEEE Transactions on Information Theory_ , vol. 53, no. 12, pp. 4457–4472, 2007.
  * [6] ——, “Error exponents for asymmetric two-user discrete memoryless source-channel systems,” in _2007 IEEE International Symposium on Information Theory_. IEEE, 2007, pp. 1736–1740.
  * [7] B. Belzer, J. D. Villasenor, and B. Girod, “Joint source channel coding of images with trellis coded quantization and convolutional codes,” in _Proceedings., International Conference on Image Processing_ , vol. 2. IEEE, 1995, pp. 85–88.
  * [8] S. Heinen and P. Vary, “Transactions papers source-optimized channel coding for digital transmission channels,” _IEEE Transactions on Communications_ , vol. 53, no. 4, pp. 592–600, 2005.
  * [9] J. Cai and C. W. Chen, “Robust joint source-channel coding for image transmission over wireless channels,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 10, no. 6, pp. 962–966, 2000.
  * [10] T. Guionnet and C. Guillemot, “Joint source-channel decoding of quasiarithmetic codes,” in _Data Compression Conference, 2004. Proceedings. DCC 2004_. IEEE, 2004, pp. 272–281.
  * [11] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, “Variable rate image compression with recurrent neural networks,” _arXiv preprint arXiv:1511.06085_ , 2015.
  * [12] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” _arXiv preprint arXiv:1611.01704_ , 2016.
  * [13] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 10 771–10 780.
  * [14] D. Minnen and S. Singh, “Channel-wise autoregressive entropy models for learned image compression,” _arXiv preprint arXiv:2007.08739_ , 2020.
  * [15] T. O’Shea and J. Hoydis, “An introduction to deep learning for the physical layer,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 3, no. 4, pp. 563–575, 2017.
  * [16] J. Xu, W. Chen, B. Ai, R. He, Y. Li, J. Wang, T. Juhana, and A. Kurniawan, “Performance evaluation of autoencoder for coding and modulation in wireless communications,” in _2019 11th International Conference on Wireless Communications and Signal Processing (WCSP)_. IEEE, 2019, pp. 1–6.
  * [17] Y. Jiang, H. Kim, H. Asnani, S. Kannan, S. Oh, and P. Viswanath, “Learn codes: Inventing low-latency codes via recurrent neural networks,” _IEEE Journal on Selected Areas in Information Theory_ , 2020.
  * [18] ——, “Turbo autoencoder: Deep learning based channel codes for point-to-point communication channels,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 2758–2768.
  * [19] E. Bourtsoulatze, D. B. Kurka, and D. Gündüz, “Deep joint source-channel coding for wireless image transmission,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 5, no. 3, pp. 567–579, 2019.
  * [20] D. B. Kurka and D. Gündüz, “Bandwidth-agile image transmission with deep joint source-channel coding,” _IEEE Transactions on Wireless Communications_ , 2021.
  * [21] ——, “Deepjscc-f: Deep joint source-channel coding of images with feedback,” _IEEE Journal on Selected Areas in Information Theory_ , vol. 1, no. 1, pp. 178–193, 2020.
  * [22] J. Xu, B. Ai, W. Chen, A. Yang, P. Sun, and M. Rodrigues, “Wireless image transmission using deep source channel coding with attention modules,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2021.
  * [23] T. Chuman, W. Sirichotedumrong, and H. Kiya, “Encryption-then-compression systems using grayscale-based image encryption for jpeg images,” _IEEE Transactions on Information Forensics and security_ , vol. 14, no. 6, pp. 1515–1525, 2019.
  * [24] W. Diffie and M. E. Hellman, “Special feature exhaustive cryptanalysis of the nbs data encryption standard,” _Computer_ , vol. 10, no. 6, pp. 74–84, 1977\.
  * [25] J. Daemen and V. Rijmen, “Reijndael: The advanced encryption standard.” _Dr. Dobb’s Journal: Software Tools for the Professional Programmer_ , vol. 26, no. 3, pp. 137–139, 2001.
  * [26] R. L. Rivest, A. Shamir, and L. Adleman, “A method for obtaining digital signatures and public-key cryptosystems,” _Communications of the ACM_ , vol. 21, no. 2, pp. 120–126, 1978.
  * [27] Z. Hui, “Arnold transformation of n dimensions and its periodicity,” _Journal of North China University of Technology_ , vol. 14, no. 1, pp. 21–25, 2002.
  * [28] K. Yano and K. Tanaka, “Image encryption scheme based on a truncated baker transformation,” _IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences_ , vol. 85, no. 9, pp. 2025–2035, 2002.
  * [29] J. Zou, R. K. Ward, and D. Qi, “A new digital image scrambling method based on fibonacci numbers,” in _2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No. 04CH37512)_ , vol. 3. IEEE, 2004, pp. III–965.
  * [30] W. Zhong, Y. H. Deng, and K. T. Fang, “Image encryption by using magic squares,” in _2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI)_. IEEE, 2016, pp. 771–775.
  * [31] R. Mathews, A. Goel, P. Saxena, and V. P. Mishra, “Image encryption based on explosive inter-pixel displacement of the rgb attributes of a pixel,” in _Proceedings of the World Congress on Engineering and Computer Science_ , vol. 1. Citeseer, 2011, pp. 19–22.
  * [32] M. Prasad, K. Sudha _et al._ , “Chaos image encryption using pixel shuffling,” _CCSEA_ , vol. 1, pp. 169–179, 2011.
  * [33] S. S. Maniccam and N. G. Bourbakis, “Image and video encryption using scan patterns,” _Pattern Recognition_ , vol. 37, no. 4, pp. 725–737, 2004.
  * [34] S. F. El-Zoghdy, Y. A. Nada, and A. Abdo, “How good is the des algorithm in image ciphering?” _International Journal of Advanced Networking and Applications_ , vol. 2, no. 5, pp. 796–803, 2011.
  * [35] B. Acharya, S. K. Panigrahy, S. K. Patra, and G. Panda, “Image encryption using advanced hill cipher algorithm,” _International Journal of Recent Trends in Engineering_ , vol. 1, no. 1, pp. 663–667, 2009.
  * [36] N. K. Pareek, V. Patidar, and K. K. Sud, “Image encryption using chaotic logistic map,” _Image and Vision Computing_ , vol. 24, no. 9, pp. 926–934, 2006.
  * [37] J. Jun, “Image encryption method based on elementary cellular automata,” in _IEEE Southeastcon 2009_. IEEE, 2009, pp. 345–349.
  * [38] Z. H. Guan, F. Huang, and W. Guan, “Chaos-based image encryption algorithm,” _Physics Letters A_ , vol. 346, no. 1-3, pp. 153–157, 2005.
  * [39] X. Tong and M. Cui, “Image encryption with compound chaotic sequence cipher shifting dynamically,” _Image and Vision Computing_ , vol. 26, no. 6, pp. 843–850, 2008.
  * [40] A. Kanso and M. Ghebleh, “A novel image encryption algorithm based on a 3d chaotic map,” _Communications in Nonlinear Science and Numerical Simulation_ , vol. 17, no. 7, pp. 2943–2959, 2012.
  * [41] B. Ferreira, J. Rodrigues, J. Leitao, and H. Domingos, “Privacy-preserving content-based image retrieval in the cloud,” in _2015 IEEE 34th symposium on reliable distributed systems (SRDS)_. IEEE, 2015, pp. 11–20.
  * [42] M. Johnson, P. Ishwar, V. Prabhakaran, D. Schonberg, and K. Ramchandran, “On compressing encrypted data,” _IEEE Transactions on Signal Processing_ , vol. 52, no. 10, pp. 2992–3006, 2004.
  * [43] W. Liu, W. Zeng, L. Dong, and Q. Yao, “Efficient compression of encrypted grayscale images,” _IEEE Transactions on Image Processing_ , vol. 19, no. 4, pp. 1097–1102, 2010.
  * [44] X. Kang, A. Peng, X. Xu, and X. Cao, “Performing scalable lossy compression on pixel encrypted images,” _EURASIP Journal on Image and Video Processing_ , vol. 2013, no. 1, pp. 1–6, 2013.
  * [45] J. Zhou, X. Liu, O. C. Au, and Y. Y. Tang, “Designing an efficient image encryption-then-compression system via prediction error clustering and random permutation,” _IEEE Transactions on Information Forensics and Security_ , vol. 9, no. 1, pp. 39–50, 2014.
  * [46] X. Zhang, “Lossy compression and iterative reconstruction for encrypted image,” _IEEE Transactions on Information Forensics and Security_ , vol. 6, no. 1, pp. 53–58, 2011.
  * [47] T. Maekawa, A. Kawamura, Y. Kinoshita, and H. Kiya, “Privacy-preserving svm computing in the encrypted domain,” in _2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2018, pp. 897–902.
  * [48] M. Tanaka, “Learnable image encryption,” in _2018 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW)_. IEEE, 2018, pp. 1–2.
  * [49] W. Sirichotedumrong, Y. Kinoshita, and H. Kiya, “Pixel-based image encryption without key management for privacy-preserving deep neural networks,” _IEEE Access_ , vol. 7, pp. 177 844–177 855, 2019.
  * [50] W. Sirichotedumrong, T. Maekawa, Y. Kinoshita, and H. Kiya, “Privacy-preserving deep neural networks with pixel-based image encryption considering data augmentation in the encrypted domain,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 674–678.
  * [51] W. Sirichotedumrong and H. Kiya, “A gan-based image transformation scheme for privacy-preserving deep neural networks,” in _2020 28th European Signal Processing Conference (EUSIPCO)_. IEEE, 2021, pp. 745–749.
  * [52] H. Ito, Y. Kinoshita, M. Aprilpyone, and H. Kiya, “Image to perturbation: An image transformation network for generating visually protected images for privacy-preserving deep neural networks,” _IEEE Access_ , vol. 9, pp. 64 629–64 638, 2021.
  * [53] A. Acar, H. Aksu, A. S. Uluagac, and M. Conti, “A survey on homomorphic encryption schemes: Theory and implementation,” _ACM Computing Surveys (CSUR)_ , vol. 51, no. 4, pp. 1–35, 2018.
  * [54] Y. Aono, T. Hayashi, L. Wang, S. Moriai _et al._ , “Privacy-preserving deep learning via additively homomorphic encryption,” _IEEE Transactions on Information Forensics and Security_ , vol. 13, no. 5, pp. 1333–1345, 2018.
  * [55] R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter, M. Naehrig, and J. Wernsing, “Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy,” in _International conference on machine learning_. PMLR, 2016, pp. 201–210.
  * [56] E. Hesamifard, H. Takabi, and M. Ghasemi, “Cryptodl: Deep neural networks over encrypted data,” _arXiv preprint arXiv:1711.05189_ , 2017.
  * [57] R. Xu, J. B. Joshi, and C. Li, “Cryptonn: Training neural networks over encrypted data,” in _2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)_. IEEE, 2019, pp. 1199–1209.
  * [58] G. K. Wallace, “The jpeg still picture compression standard,” _IEEE Transactions on Consumer Electronics_ , vol. 38, no. 1, pp. xviii–xxxiv, 1992\.
  * [59] M. Rabbani, “Jpeg2000: Image compression fundamentals, standards and practice,” _Journal of Electronic Imaging_ , vol. 11, no. 2, p. 286, 2002\.
  * [60] K. Xu, M. Qin, F. Sun, Y. Wang, Y.-K. Chen, and F. Ren, “Learning in the frequency domain,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1740–1749.
  * [61] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [62] N. Farsad, M. Rao, and A. Goldsmith, “Deep learning for joint source-channel coding of text,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 2326–2330.
  * [63] K. Choi, K. Tatwawadi, A. Grover, T. Weissman, and S. Ermon, “Neural joint source-channel coding,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 1182–1192.
  * [64] Y. M. Saidutta, A. Abdi, and F. Fekri, “Joint source-channel coding for gaussian sources over awgn channels using variational autoencoders,” in _2019 IEEE International Symposium on Information Theory (ISIT)_. IEEE, 2019, pp. 1327–1331.
  * [65] ——, “Joint source-channel coding of gaussian sources over awgn channels via manifold variational autoencoders,” in _2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)_. IEEE, 2019, pp. 514–520.
  * [66] F. Jiang, W. Tao, S. Liu, J. Ren, X. Guo, and D. Zhao, “An end-to-end compression framework based on convolutional neural networks,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 28, no. 10, pp. 3007–3018, 2018.
  * [67] D. Mishra, S. K. Singh, and R. K. Singh, “Wavelet-based deep auto encoder-decoder (wdaed)-based image compression,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 4, pp. 1452–1462, 2021\.
  * [68] P. Mohammadi, A. Ebrahimi-Moghadam, and S. Shirani, “Subjective and objective quality assessment of image: A survey,” _arXiv preprint arXiv:1406.7799_ , 2014.
  * [69] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European Conference on Computer Vision_. Springer, 2016, pp. 694–711.
  * [70] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [71] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin _et al._ , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” _arXiv preprint arXiv:1603.04467_ , 2016.
  * [72] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [73] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [74] A. H. Chang and B. M. Case, “Attacks on image encryption schemes for privacy-preserving deep neural networks,” _arXiv preprint arXiv:2004.13263_ , 2020.
  * [75] W. Sirichotedumrong and H. Kiya, “Visual security evaluation of learnable image encryption methods against ciphertext-only attacks,” in _2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2020, pp. 1304–1309.


11institutetext:  Department of Physics and Astronomy,
University of Sussex, Brighton, BN1 9QH, United Kingdom
Department of Physics and Astronomy
Michigan State University, East Lansing, Michigan 48823, USA

# A Brief History of Hawking’s Information Paradox

Xavier Calmet E-mail<EMAIL_ADDRESS>11 Stephen D. H. Hsu E-mail:
<EMAIL_ADDRESS>221122

###### Abstract

In this invited review, we describe Hawking’s information paradox and a
recently proposed resolution of it. Explicit calculations demonstrate the
existence of quantum hair on black holes, meaning that the quantum state of
the external graviton field depends on the internal state of the black hole.
Simple quantum mechanics then implies that Hawking radiation amplitudes depend
on the internal state, resulting in a pure final radiation state that
preserves unitarity and, importantly, violates a factorization assumption
which is central to the original paradox. Black hole information is encoded in
entangled macroscopic superposition states of the radiation.

##  1 Introduction and Formulation of Paradox

The thermal nature of his radiation led Stephen Hawking in 1976 to argue that
black holes destroy quantum information [1]111 Given the length limit for a
Perspective article, we will not be able to review all the work since
Hawking’s seminal paper: see e.g.[2, 3, 4, 5, 6, 7, 8]. Rather, we will focus
on the material that is directly relevant to recent advances involving quantum
hair.. More precisely, Hawking argued that black holes cause pure states to
evolve into mixed states. Quantum information that falls into a black hole
does not escape in the form of radiation. Rather, it vanishes completely from
our universe, thereby violating unitarity in quantum mechanics.

In 2009 Mathur [9, 10] gave an especially clear formulation of the paradox.
His analysis tracks the entanglement entropy of Hawking radiation emitted on a
nice slice. Nice slices are spacelike surfaces which intersect both the
interior of the black hole and the emitted Hawking radiation. Mathur
emphasizes how the Hawking radiation originates from vacuum fluctuations of
entangled particle and antiparticle states. The vacuum state near the horizon
of a large black hole (i.e., where curvature is negligible) must be
approximately that of the ordinary vacuum. One of the entangled pair (the
Hawking radiation) escapes to infinity, while the other particle falls into
the black hole. When the black hole finally evaporates and disappears from our
universe, the radiation modes are left in a mixed state with very large
entanglement entropy.

Let us call the modes outside the horizon $b_{1},b_{2},...$, and those inside
the horizon $e_{1},e_{2},...$. The initial slice in the nice slice foliation
contains only the matter state $|\psi\rangle_{M}$ (i.e., the black hole), and
none of the $e_{i},b_{i}$. The first step of evolution stretches the spacelike
slice, so that the particle modes $e_{1},b_{1}$ are now present on the new
slice. The state of these modes has the schematic form

$\frac{1}{\sqrt{2}}\Big{(}|0\rangle_{e_{1}}|0\rangle_{b_{1}}+|1\rangle_{e_{1}}|1\rangle_{b_{1}}\Big{)}~{},$
(1)

where the numbers $0,1$ give the occupation number of a particle mode. The
entanglement of the state outside the horizon (given by the mode $b_{1}$) with
the state inside (given by the mode $e_{1}$) yields $S_{\rm entanglement}=\ln
2$.

In the next step of evolution the modes $b_{1},e_{1}$ at the earlier step move
apart, and in the region between them there appears another pair of modes
$b_{2},e_{2}$ in a state that has the same form as (1). After $N$ steps we
have the state

$\displaystyle|\Psi\rangle\approx~{}|\psi\rangle_{M}$ $\displaystyle\otimes$
$\displaystyle\frac{1}{\sqrt{2}}\Big{(}|0\rangle_{e_{1}}|0\rangle_{b_{1}}+|1\rangle_{e_{1}}|1\rangle_{b_{1}}\Big{)}$
(2) $\displaystyle\otimes$
$\displaystyle\frac{1}{\sqrt{2}}\Big{(}|0\rangle_{e_{2}}|0\rangle_{b_{2}}+|1\rangle_{e_{2}}|1\rangle_{b_{2}}\Big{)}$
(3) $\displaystyle\dots$ (4) $\displaystyle\otimes$
$\displaystyle\frac{1}{\sqrt{2}}\Big{(}|0\rangle_{e_{N}}|0\rangle_{b_{N}}+|1\rangle_{e_{N}}|1\rangle_{b_{N}}\Big{)}.~{}$
(5)

The state $\Psi$ has the tensor product form given above because each pair
creation (Hawking emission) is widely separated from the others. There is no
connection to the matter state in the leading order Hawking process. It is the
factorized form of $\Psi$ that leads to the paradox, once the black hole and
$e_{i}$ modes are gone. This formulation has been further developed in [11,
12], though it is fair to say that all formulations of the paradox are based
on assumptions related to the factorization of states apparent in (5).

The modes $b_{i}$ are entangled with the $e_{i}$ and the black hole with
$S_{\rm entanglement}=N\ln 2$. This entanglement grows by $\ln 2$ with each
succeeding emission. If the black hole evaporates away completely, the $b_{i}$
quanta outside will be in an entangled state, but there will be nothing that
they are entangled with. The initial pure state has evolved to a mixed state,
described by a density matrix.

Mathur argues further that small corrections $\epsilon$ to Hawking evaporation
cannot change this qualitative result: the entanglement entropy increases by
$\approx\ln 2-\epsilon$ with each emitted quantum. We note, however, that this
entire analysis has been specific to a given spacetime geometry which defines
(on each nice slice) the location of the black hole horizon, the nature of the
$e_{i}$ modes, etc. Nowhere in this analysis has the possibility been
considered that there might be other background geometries involved: e.g.,
with black hole center of mass at a different location, or with different
recoil velocity, or equivalently with a macroscopically different pattern of
radiation $\\{b_{1},b_{2},...,b_{N}\\}$ emitted. The black hole trajectory is
determined by recoil from the emitted radiation, so different patterns
correspond to different trajectories. Mathur’s analysis does not consider
whether small $\epsilon$ entanglements between macroscopically distinct
radiation states can unitarize the evolution of $\Psi$. This will be examined
further below.

##  2 AdS/CFT and Holography

The holographic principle – i.e., that bulk information in models of gravity
in $d$-dimensions might be available on the $(d-1)$ dimensional boundary of
spacetime – is due to ’t Hooft (1993) [13] and Susskind (1995) [14].
Holography was given an explicit realization in the AdS/CFT correspondence of
Maldacena (1998) [15], which suggests that black hole evaporation can be
unitary. That is, formation and evaporation of a small black hole in the bulk
is dual to the (presumably unitary) evolution of some gauge configuration on
the boundary. Indeed, as conventionally interpreted, the AdS/CFT
correspondence provides a wide range of examples of unitary quantum dynamics
of black holes. Recently there has been considerable progress in directly
computing the entanglement entropy of evaporation using AdS methods222Under
Schrödinger evolution the quantum state describing a small black hole in AdS
will become a macroscopic superposition state with the hole on different
evaporation trajectories. Presumably the gauge dual of this bulk state evolves
similarly. Neither outcome is surprising, given the results in [16]. It would
be interesting to explore whether these observations are reflected in bulk-
boundary duality [17]., and these results suggest that the process is unitary
[17].

However, the precise mechanism by which black hole information is encoded in
outgoing radiation quanta has never been identified. To fully resolve the
paradox requires an understanding of how the bulk gravitational dynamics is
modified so that it becomes unitary. Various scenarios have been advocated
over time, including exotic scenarios with gross violations of locality near
the horizon, or stable black hole remnants. It is unclear whether exotic new
physics is required to resolve the paradox – our analysis below suggests that
it is not.

Recently, the authors of [18, 19, 20, 21] have argued that bulk information is
recoverable near the boundary of spacetime via local measurements. They do not
assume a specific short distance completion of quantum gravity, but make some
plausible assumptions, such as that the spectrum of energy eigenvalues is
bounded below. If correct, these results provide a concrete realization of
holography that the authors refer to as holography of information.

Holography of information implies that the internal quantum state of a black
hole must be encoded in the asymptotic quantum state of its graviton field,
since otherwise the information would not be recoverable at the boundary.
However, the details of how this works do not appear in [18, 19, 20, 21]. The
notion of quantum hair that we introduce below enables us to address this
question.

The idea that information cannot be localized in quantum gravity, and hence is
recoverable at the boundary of spacetime, can be regarded as a consequence of
the Gauss Law constraints on quantization, combined with the fact that mass
(i.e., gravitational charge) cannot be screened333Absence of gravitational
screening in the low-energy effective theory of gravity is commonly assumed,
although mathematical studies of exotic (perhaps unphysical) spacetimes may
suggest otherwise [24]. We show explicitly [28, 27] that information about the
black hole internal state is available in the quantum state of its gravity
field (quantum hair).. Note this assumes the absence of negative mass states –
i.e., a lower bound on the energy spectrum in quantum gravity. An alternative
argument can be given using diffeomorphism invariance, which implies the
absence of local observables in gravitational theories, see e.g. [22, 23] and
references therein.

While both holography of information and AdS/CFT suggest that the Hawking
paradox is somehow resolved in favor of unitarity, neither yield a specific
description of the physical process by which black hole information is encoded
in Hawking radiation which originates outside the horizon. This question has
remained a mystery over the 20+ years that have elapsed since Maldacena gave
the first examples of bulk-boundary duality. As evidence supporting this
conclusion, see these recent reviews [25, 26].

##  3 Quantum Hair and Unitary Evaporation

We shall now describe the solution to Hawking’s paradox proposed in [27]. In
[28], the analysis of the state of the graviton field produced by a compact
matter source (e.g., a black hole) revealed the following:

1\. The asymptotic graviton state of an energy eigenstate source is determined
at leading order by the energy eigenvalue, and can be expressed explicitly as
a coherent state which depends on this eigenvalue. Insofar as there are no
accidental energy degeneracies there is a one to one map between graviton
states and matter source states444It is a folk theorem in many-body physics
that typical energy level splittings are $\sim\exp(-S)$ ($S$ is the entropy of
the system) and there are no exact degeneracies without exact symmetries. Even
if distinct but exactly degenerate energy eigenstates $\psi_{1}$ and
$\psi_{2}$ exist, it still seems likely that they are distinguishable via
their effects on the graviton quantum state since by assumption the functions
$\psi_{1}(x)$ and $\psi_{2}(x)$ are not identical. These effects are
subleading relative to the dependence on the energy eigenvalue.. A
semiclassical matter source produces an entangled graviton state.

2\. Quantum gravitational fluctuations (i.e., graviton loops) produce
corrections to the long range potential (e.g., $\sim r^{-5}$) whose
coefficients depend on the internal state of the source. This provides an
explicit example of how the graviton quantum state (corresponding to the
semiclassical potential) encodes information about the internal state of a
black hole. Note the calculation is insensitive to short distance effects in
quantum gravity (i.e., the short distance completion of the model).

Using these properties, we can write the quantum state of the exterior metric
(equivalently, the quantum state of the exterior geometry) as

$\Psi_{i}=\sum_{n}c_{n}\Psi_{g}(E_{n})=\sum_{n}c_{n}\,|\,g(E_{n})\,\rangle~{}.$
(6)

A semiclassical state has support concentrated in some range of energies $E$,
where the magnitudes of $c_{n}$ are largest. For simplicity, when representing
the exterior metric state $g(E)$ we only write the energy explicitly and
suppress the other quantum numbers.

Assume for convenience that the black hole emits one quantum at a time (e.g.,
at fixed intervals), culminating in a final state of $N$ radiation quanta:

$|~{}r_{1}~{}r_{2}~{}r_{3}~{}\cdots~{}r_{N}~{}\rangle~{}.$

The quantum numbers of the $i$-th emitted radiation particle include the
energy $\Delta_{i}$, momentum $p_{i}$, spin $s_{i}$, charge $q_{i}$, etc. The
symbol $r_{i}$ is used to represent all of these values:

$r_{i}\sim\\{\Delta_{i},p_{i},s_{i},q_{i},\ldots\\}~{}.$

A final radiation state is specified by the values of
$\\{r_{1},r_{2},\ldots,r_{N}\\}$.

Let the amplitude for emission of quantum $r_{i}$ from exterior metric state
$\Psi_{g}(E)$ be $\alpha(E,r_{i})$. This amplitude must approximate the
semiclassical Hawking amplitude for a black hole of mass $E$. In the leading
approximation the amplitudes are those of thermal emission, but at subleading
order (i.e., $\sim S^{-k}$ for perturbative corrections such as those
calculated in [28], or $\exp(-S)$ for nonperturbative effects, where $S$ is
the black hole entropy) additional dependence on $(E,r_{i})$ will emerge. The
fact that these corrections can depend on the internal state of the hole is a
consequence of quantum hair. It has been shown that even corrections as small
as $\exp(-S)$ can purify a maximally mixed Hawking state (i.e., can perturb
the radiation density matrix $\rho$ so that ${\rm tr}\,\rho^{2}=1$), because
the dimensionality ($\sim\exp S$) of the Hilbert space is so large [29, 30,
31, 32].

When the black hole emits the first radiation quantum $r_{1}$ it evolves into
the exterior state given on the right below:

$\Psi_{i}~{}\rightarrow~{}\sum_{n}\sum_{r_{1}}c_{n}~{}\alpha(E_{n},r_{1})~{}|\,g(E_{n}-\Delta_{1}),r_{1}\rangle~{}~{}.$
(7)

In this notation $g$ refers to the exterior geometry and $r_{1}$ to the
radiation. The next emission leads to

$\displaystyle\sum_{n}\sum_{r_{1},r_{2}}c_{n}~{}\alpha(E_{n},r_{1})~{}\alpha(E_{n}-\Delta_{1},r_{2})\times$
$\displaystyle|\,g(E_{n}-\Delta_{1}-\Delta_{2}),r_{1},r_{2}\rangle~{}~{},$ (8)

and the final radiation state is

$\displaystyle\sum_{n}\sum_{r_{1},r_{2},\ldots,r_{N}}c_{n}~{}\alpha(E_{n},r_{1})~{}\alpha(E_{n}-\Delta_{1},r_{2})~{}\times$
$\displaystyle\alpha(E_{n}-\Delta_{1}-\Delta_{2},r_{3})~{}\cdots~{}|\,r_{1}~{}r_{2}~{}\cdots~{}r_{N}\rangle~{}.$
(9)

In the final state we omit reference to the geometry $g$ as the black hole no
longer exists: there is no horizon and the spacetime is approximately flat.

As is the case with the decay of any macroscopic object into individual
quanta, several remarks apply. A semiclassical black hole state has support
concentrated in a small range of energy eigenvalues $E_{n}$. Realistic
radiation quanta, which are localized in space and time, are better
represented as wave packet states than as plane wave states, and hence have a
small spread as well in their energies $\Delta_{i}$. The energy conservation
requirement: $E_{n}\approx\sum_{i}\,\Delta_{i}$ is not exact.

The final expression (3) is the same one we would obtain from a burning lump
of coal if we interpret the amplitudes $\alpha(E,r)$ with $r$ (again) the
state of the radiated particle and $E$ the state of the coal as it burns.

Quantum hair allows the internal state of the black hole, reflected in the
coefficients $c_{n}$, to affect the Hawking radiation. The result is
manifestly unitary, and the final state in (3) is manifestly a pure state:

$\bullet$ For each distinct initial state given by the $\\{c_{n}\\}$ there is
a different final radiation state.

$\bullet$ The time-reversed evolution of a final radiation state results in a
specific initial state.

Note the final radiation state in (3) is a macroscopic superposition state.
Configurations described by $\\{r_{1},r_{2},\ldots,r_{N}\\}$ will exhibit
different regions with higher or lower densities of energy, charge, etc. Time-
reversed evolution will only produce the original semiclassical black hole
state if these radiation states converge and interfere with the exact phase
relations given in (3).

We emphasize that the quantum hair provides a mechanism by which the
amplitudes $\alpha(E,r)$ can depend on the internal state. These results
provide, for the first time, a physical picture of how the information is
encoded in the Hawking radiation.

Specifically, the amplitudes $\alpha(E,r)$ can be explicitly calculated using
the quantum corrected metric derived in [28, 33] using effective field theory
methods. We expect a correction to Hawking’s thermal radiation spectrum which
is dependent on the composition of the original star via the corrections
encoded in the $G_{N}^{2}r^{-5}$ quantum gravitational corrections to the
potential. This explicitly shows how Hawking radiation retains a memory of the
original matter configuration that collapsed to a black hole.

##  4 Mathur’s Theorem revisited

Recall that in Mathur’s formulation of the paradox each vacuum fluctuation
contains an entangled pair $(e_{k},b_{k})$, with the $b_{k}$ modes escaping to
infinity as Hawking radiation, and the $e_{k}$ modes falling into the black
hole. These modes are defined with respect to a specific background geometry
which has to be consistent with the overall coarse grained pattern of emitted
radiation.

Two radiation patterns $\\{b_{1},b_{2},...,b_{N}\\}$ and
$\\{b^{\prime}_{1},b^{\prime}_{2},...,b^{\prime}_{N}\\}$ which are
sufficiently different correspond to distinct semiclassical geometries and
hence distinct nice slices. The existence of distinct geometries becomes
apparent as soon as we realize that the black hole recoil trajectory during
evaporation is determined by the radiation pattern. The set of possible recoil
trajectories is a collection of random walks, and the eventual spread in
location of the hole is macroscopic, of order $\sim M^{2}$.

This formulation of the paradox does not address the possibility of
entanglement between different geometries (i.e., between sufficiently
different patterns of radiation as described above). Furthermore, the nature
of the infalling $e$ modes is different on each geometry as they are
excitations with respect to a specific nice slice.

The $b$ modes are the $r$ modes in Eq. (3). In our description the
entanglement between $b$ and $e$ modes is reflected in the entanglement
between $r$ modes and the exterior metric state $g(E)$, which is itself
determined by the internal black hole state.

A negative energy $e$ mode reduces the total energy of the internal state:
$E\rightarrow E-\Delta$, and this is then reflected in the exterior metric
state $g(E-\Delta)$. Other quantum numbers such as spin or charge which
characterize the interior state are similarly modified by the infalling $e$
modes.

The analog of the entangled state
$\Big{(}|0\rangle_{e_{i}}|0\rangle_{b_{i}}+|1\rangle_{e_{i}}|1\rangle_{b_{i}}\Big{)}$
is simply

$|\,g(E-\Delta),r\,\rangle~{}+~{}|\,g(E-\Delta^{\prime}),r^{\prime}\,\rangle~{}~{}.$
(10)

Upon inspection, it is easy to see that our final radiation state (3) contains
many “copies” of the Mathur state, corresponding to different background
geometries, which are themselves determined by the coarse grained stress
tensor of a specific radiation pattern. However, our expression is manifestly
unitary and exhibits entanglements across different background geometries, or
equivalently between radiation patterns that differ macroscopically. These
entanglements do not appear anywhere in Mathur’s formulation and are
responsible for unitarization of the evaporation process.

##  5 Pure vs Mixed States and Concentation of Measure

Let $Y$ be the Hilbert space describing all possible radiation patterns
$\\{r_{1},r_{2},\cdots r_{N}\\}$. The subset of radiation patterns $X$ which
is consistent with a specific semiclassical geometry (i.e., nice slice) is a
very small subset $X$ of the final state Hilbert space $Y$.

Concentration of measure (Levy’s lemma) [16] implies that for almost all pure
states $\Psi$ in a large Hilbert space $Y$, the density matrix

$\rho(X)={\rm tr}~{}\Psi^{\dagger}\,\Psi$ (11)

describing a small subset $X$ (tracing over the complement of $X$ in $Y$) is
nearly maximally mixed – i.e., like the radiation found in Hawking’s original
calculation. That is, a pure final state in $Y$ resulting from black hole
evaporation might appear to be nearly maximally mixed to an observer (i.e.,
measuring devices) restricted to a specific, fixed background geometry.

The fact that pure states in large Hilbert spaces are very difficult to
distinguish from mixed states is central to the idea of decoherence:
exponentially sensitive measurements are required to determine whether pure
states evolve into mixed states (as in the case of wavefunction collapse) or
merely appear to (decoherence) [34].

In almost all discussions of the black hole information paradox, entanglement
between different $X$ and $X^{\prime}$ (equivalently, between different
branches of the wavefunction $\Psi$) has been neglected, although even
exponentially small entanglement between these branches may be sufficient to
unitarize the result [29, 30, 31].

The final state in Eq. (3)

$\displaystyle\sum_{n}\sum_{r_{1},r_{2},\ldots,r_{N}}c_{n}~{}\alpha(E_{n},r_{1})~{}\alpha(E_{n}-\Delta_{1},r_{2})~{}$
$\displaystyle\alpha(E_{n}-\Delta_{1}-\Delta_{2},r_{3})~{}\cdots~{}|\,r_{1}~{}r_{2}~{}\cdots~{}r_{N}\rangle$

exhibits such $X$-$X^{\prime}$ entanglements: it does not factorize across
semiclassical spacetimes.

As we noted previously, even corrections as small as $\exp(-S)$ can purify a
maximally mixed Hawking state (i.e., can perturb a maximally mixed radiation
density matrix $\rho$ so that ${\rm tr}\,\rho^{2}=1$), because the
dimensionality of the Hilbert space $Y$ ($\sim\exp S$) is so large.

##  6 Final Remarks

Consider a burning lump of coal instead of an evaporating black hole.
Following a similar analysis we obtain an expression for the final radiation
state like Eq. (3) except that $\alpha(E,r)$ is the amplitude for the coal
state with energy eigenvalue $E$ to emit radiation state $r$. Due to
fluctuations in the (coarse grained) temperature and pressure of the emitted
radiation, the initial coal state also evolves into a macroscopic
superposition of radiation states. This is not surprising: it has been shown
using von Neumann’s 1929 Quantum Ergodic Theorem [35] that under very weak
assumptions generic initial states will evolve into macroscopic superposition
states [16].

Because our expressions (7)-(3) are manifestly unitary, and the final
radiation state is pure, the Page curve [36, 37] of entanglement entropy as a
function of evaporation time returns to zero. The entanglement entropy of the
radiation state increases from zero, reaches a maximum while the evaporation
is in progress, and returns to zero when the black hole has vanished and only
the final pure state (3) remains.

The radiation amplitudes computed by Hawking, which describe thermal radiation
emitted from a black hole at temperature $T$, already describe a broad
distribution of possible radiation types, spins, and momenta emitted at each
stage. Thus, even in the semiclassical approximation there are many distinct
patterns of radiation in (3). The set of possible final states is already
complex even at leading order, resulting in very different coarse grained
patterns of energy-momentum density. Small corrections to the amplitudes
$\alpha(E,r)$ due to quantum hair do not qualitatively change this situation,
but they are necessary to unitarize the evaporation and they determine the
precise relations between components of the entangled state.

Importantly, no special assumptions about the amplitudes $\alpha(E,r)$ need to
be made to determine that the factorized form of the state (5) does not hold.
Factorization is assumed in essentially every formulation of the information
paradox, but in reality is violated because the external graviton state
depends on the internal black hole state.

Known quantum gravitational effects leading to quantum hair are extremely
small and thus difficult to probe experimentally or detect via observations.
We cannot prove that our solution to the information paradox is unique.
However, the consequences of quantum hair lead, without any speculative
theoretical assumptions, to plausible unitary evaporation of black holes. The
properties of quantum hair and the evaporation amplitude (3) can be deduced
using only long wavelength properties of quantum gravity – they do not rely on
assumptions about Planck scale physics or a specific short distance
completion. Therefore, Occam’s razor favors quantum hair.

###### Acknowledgements.

The work of X.C. is supported in part by the Science and Technology Facilities
Council (grants numbers ST/T00102X/1 and ST/T006048/1). Data Availability
Statement: This manuscript has no associated data. Data sharing not applicable
to this article as no datasets were generated or analysed during the current
study.

## References

  * [1] S. W. Hawking, Phys. Rev. D 13 (1976), 191-197 doi:10.1103/PhysRevD.13.191.
  * [2] G. ’t Hooft, Nucl. Phys. B Proc. Suppl. 43 (1995), 1-11 doi:10.1016/0920-5632(95)00444-E
  * [3] S. W. Hawking, Phys. Rev. D 72 (2005), 084013 doi:10.1103/PhysRevD.72.084013 [arXiv:hep-th/0507171 [hep-th]].
  * [4] S. W. Hawking, [arXiv:1401.5761 [hep-th]].
  * [5] S. W. Hawking, [arXiv:1509.01147 [hep-th]].
  * [6] S. B. Giddings, [arXiv:hep-th/9508151 [hep-th]].
  * [7] R. Bousso, X. Dong, N. Engelhardt, T. Faulkner, T. Hartman, S. H. Shenker and D. Stanford, [arXiv:2201.03096 [hep-th]].
  * [8] N. Engelhardt and A. Folkestad, Phys. Rev. D 105 (2022) no.8, 086010 doi:10.1103/PhysRevD.105.086010 [arXiv:2201.08395 [hep-th]].
  * [9] S. D. Mathur, Class. Quant. Grav. 26 (2009), 224001 doi:10.1088/0264-9381/26/22/224001 [arXiv:0909.1038 [hep-th]].
  * [10] S. D. Mathur, [arXiv:1108.0302 [hep-th]].
  * [11] A. Almheiri, D. Marolf, J. Polchinski and J. Sully, “Black Holes: Complementarity or Firewalls?,” JHEP 02 (2013), 062 doi:10.1007/JHEP02(2013)062 [arXiv:1207.3123 [hep-th]].
  * [12] A. Almheiri, D. Marolf, J. Polchinski and J. Sully, “An Apologia for Firewalls,” JHEP 09 (2013), 018 doi:10.1007/JHEP09(2013)018 [arXiv:1304.6483 [hep-th]].
  * [13] G. ’t Hooft, Conf. Proc. C 930308 (1993), 284-296 [arXiv:gr-qc/9310026 [gr-qc]].
  * [14] L. Susskind, J. Math. Phys. 36, 6377-6396 (1995) doi:10.1063/1.531249 [arXiv:hep-th/9409089 [hep-th]].
  * [15] J. M. Maldacena, Adv. Theor. Math. Phys. 2, 231-252 (1998) doi:10.1023/A:1026654312961 [arXiv:hep-th/9711200 [hep-th]].
  * [16] R. V. Buniy and S. D. H. Hsu, Found. Phys. 51 (2021) no.4, 85 doi:10.1007/s10701-021-00477-2 [arXiv:2011.11661 [quant-ph]].
  * [17] A. Almheiri, T. Hartman, J. Maldacena, E. Shaghoulian and A. Tajdini, Rev. Mod. Phys. 93 (2021) no.3, 035002 doi:10.1103/RevModPhys.93.035002 [arXiv:2006.06872 [hep-th]].
  * [18] A. Laddha, S. G. Prabhu, S. Raju and P. Shrivastava, SciPost Phys. 10, no.2, 041 (2021) doi:10.21468/SciPostPhys.10.2.041 [arXiv:2002.02448 [hep-th]].
  * [19] C. Chowdhury, O. Papadoulaki and S. Raju, SciPost Phys. 10, no.5, 106 (2021) doi:10.21468/SciPostPhys.10.5.106 [arXiv:2008.01740 [hep-th]].
  * [20] C. Chowdhury, V. Godet, O. Papadoulaki and S. Raju, [arXiv:2107.14802 [hep-th]].
  * [21] S. Raju, Phys. Rept. 943 (2022), 1-80 doi:10.1016/j.physrep.2021.10.001 [arXiv:2012.05770 [hep-th]].
  * [22] S. B. Giddings, D. Marolf and J. B. Hartle, Phys. Rev. D 74 (2006), 064018 doi:10.1103/PhysRevD.74.064018 [arXiv:hep-th/0512200 [hep-th]].
  * [23] C. Kiefer, “Quantum Gravity,” Third edition, Oxford Univ. Press, Oxford 2012, pp 432, 10.1093/acprof:oso/9780199585205.001.0001.
  * [24] A. Carlotto and R. Schoen, Invent. math. 205, 559–6.
  * [25] I. Bena, E. J. Martinec, S. D. Mathur and N. P. Warner, “Snowmass White Paper: Micro- and Macro-Structure of Black Holes,” [arXiv:2203.04981 [hep-th]].
  * [26] E. J. Martinec, [arXiv:2203.04947 [hep-th]].
  * [27] X. Calmet and S. D. H. Hsu, Phys. Lett. B 827 (2022), 136995 doi:10.1016/j.physletb.2022.136995 [arXiv:2112.05171 [hep-th]].
  * [28] X. Calmet, R. Casadio, S. D. H. Hsu and F. Kuipers, Phys. Rev. Lett. 128 (2022) no.11, 111301 doi:10.1103/PhysRevLett.128.111301 [arXiv:2110.09386 [hep-th]].
  * [29] K. Papadodimas and S. Raju, JHEP 10, 212 (2013) doi:10.1007/JHEP10(2013)212 [arXiv:1211.6767 [hep-th]].
  * [30] S. D. H. Hsu, [arXiv:1302.0451 [hep-th]].
  * [31] S. D. H. Hsu, [arXiv:1308.5686 [hep-th]].
  * [32] N. Bao, S. M. Carroll, A. Chatwin-Davies, J. Pollack and G. N. Remmen, Phys. Rev. D 97, no.12, 126014 (2018) doi:10.1103/PhysRevD.97.126014 [arXiv:1712.04955 [hep-th]].
  * [33] X. Calmet, R. Casadio and F. Kuipers, Phys. Rev. D 100 (2019) no.8, 086010 doi:10.1103/PhysRevD.100.086010 [arXiv:1909.13277 [hep-th]].
  * [34] S. D. H. Hsu and D. Reeb, Phys. Rev. D 79 (2009), 124037 doi:10.1103/PhysRevD.79.124037 [arXiv:0903.2258 [gr-qc]].
  * [35] J. von Neumann, Z. Physik 57, 30–70 (1929).
  * [36] D. N. Page, Phys. Rev. Lett. 71 (1993), 3743-3746 doi:10.1103/PhysRevLett.71.3743 [arXiv:hep-th/9306083 [hep-th]].
  * [37] D. N. Page, JCAP 09 (2013), 028 doi:10.1088/1475-7516/2013/09/028 [arXiv:1301.4995 [hep-th]].