# Attention Mechanism for Contrastive Learning in GAN-based
Image-to-Image Translation

###### Abstract

Using real road testing to optimize autonomous driving algorithms is time-
consuming and capital-intensive. To solve this problem, we propose a GAN-based
model that is capable of generating high-quality images across different
domains. We further leverage Contrastive Learning to train the model in a
self-supervised way using image data acquired in the real world using real
sensors and simulated images from 3D games. In this paper, we also apply an
Attention Mechanism module to emphasize features that contain more information
about the source domain according to their measurement of significance.
Finally, the generated images are used as datasets to train neural networks to
perform a variety of downstream tasks to verify that the approach can fill in
the gaps between the virtual and real worlds.

Index Terms—  GAN, Self-supervised, Attention Mechanism, Contrastive Learning

##  1 Introduction

With the development of deep learning, rapid progress has been made in
autonomous driving technology. However, training the deep learning model needs
a massive amount of labeled real road data. Autonomous driving vehicles also
require to go through a lot of road testing before they can actually be used
in transportation. Using real road testing to optimize autonomous driving
algorithms is time-consuming and capital-intensive, and testing on open roads
has safety implications and regulatory constraints. In addition, extreme
weather conditions and complex traffic scenarios are difficult to be
reproduced in reality. Therefore, it makes sense to convert simulator image
data into photo-realistic image data, which can save time and capital costs
and help the research and implementation of autonomous driving algorithms.

(a) High-poly 3D Models

(b) High Definition Rendering

(c) GAN-based Post-processing

Fig. 1: Three Steps towards Photo-realistic Virtualization of Road Object

Figure 1 shows the three steps toward Photo-realistic Virtualization of road
objects. In this paper, we focus on solving the third step towards the photo-
realistic virtualization of road objects, namely the GAN-based post-processing
algorithms. Specifically, we need to design a suitable generator that can
learn the joint distribution of the two domains (real and simulated world) and
find transformations between them. Considering the difficulty of acquiring
paired images, i.e. generated labels, we need to train the generator with
unpaired images for self-supervised learning. Moreover, in order to maximize
the mutual information between the source and the translated images across
different domains, we also need to solve the problem that Generators need to
be able to identify important information and features.

The main contributions of this paper can be summarized as follows:

  * •

We introduce the main modules and Attention Mechanism module into our
Attention-based CUT_GAN. Then we evaluate the performance of the designed
model on several datasets to check its validity.

  * •

We compare several attention mechanisms through ablation experiments to verify
the improvement of different attention mechanisms on the performance of
generators. Furthermore, the effectiveness of the attention mechanism module
is verified by comparing the designed model with other generative algorithms.

##  2 Related works

In this section, we make a comprehensive review of the development of the
methods used in this paper, including Generative Adversarial Networks, Image-
to-Image Generators, and self-supervised methods.

###  2.1 Generative Adversarial Networks

Since Ian GoodFellow proposed the GAN model [1] in 2014, generative
adversarial networks have rapidly become the hottest generative model. CGAN
(conditional GAN)[2] adds constraints to the original GANs so that the network
can generate samples in a given direction. The proposal of DCGAN (deep
convolutional GAN)[3] has great significance to the development of GAN, which
combines convolutional neural networks (CNN) and GAN. It combines CNN and GAN
to ensure the quality and diversity of the generated images. LSGAN (least
squares GAN)[4], also known as least squares generative adversarial network,
replaces the cross-entropy loss function of a traditional GAN with a least
squares loss function, which effectively improves the problems of low quality
and unstable training of the original GANs. CycleGAN[5] allows images from two
domains to be transformed into each other and does not require paired images
as training data. BigGAN[6] gets its performance boost by scaling up the
model, and is so far the best GAN model for generating image quality.

###  2.2 Image-to-image Translation

The image-to-image translation methods can be classified into two classes:
Supervised methods and Unsupervised methods.

Supervised methods such as Pix2pix[7] can translate images from one domain to
another e.g. from night to day, from sketch to real picture and so on, but
images must be paired.

In unsupervised methods, Cycle GAN[5] first translates an image from a source
domain A to a target domain B in the absence of paired examples, then couples
it with an inverse mapping: domain B to domain A and uses cycle consistency
loss to let generated A image the same as the original image as much as
possible. In ACGAN[8], the discriminator not only discriminates whether the
input image is from the distribution of the generated data or from the
distribution of the real data, but also predicts the class of the input image.

###  2.3 Self-supervised Learning

Self-supervised learning differs from unsupervised learning mainly in that
unsupervised learning focuses on detecting specific data patterns, such as
clustering, community discovery, or anomaly detection, while self-supervised
learning aims at recovering and remaining within the supervised learning
paradigm.

According to Jie Tang et al.[9], self-supervised learning can be divided into
three general categories: Generative, Contrastive, and Generative-Contrastive.
ContraGAN[10], which uses Contrastive Learning, tries to draw close to image
representations that have the same category label, when considering drawing
close to the same transformed image.

##  3 Attention$-$based CUT_GAN

In this section, we introduce the structure of our proposed Image-to-Image
Translation Network called Attention-based CUT_GAN. Sections 3.2 and 3.3
introduce the two parts of Generative adversarial networks: Generator and
Discriminator. Section 3.4 describes Patchwise Contrastive Learning and other
parts of the loss function.

###  3.1 Network Overview

The overflow of our proposed Image-to-Image Translation Network is depicted in
figure 2. We wish to translate images from input domain $\chi\subset
R^{H\times W\times 3}$ to appear like an image from the output domain
$Y\subset R^{H\times W\times 3}$. We are given a dataset of unpaired instances
$X=$ {$x\in\chi$}, $Y=${$y\in Y$}. Our method only requires learning the
mapping in one direction and avoids using inverse auxiliary generators and
discriminators. This can largely simplify the training procedure and reduce
training time.

Fig. 2: Overview of the proposed Image-to-Image Translation Network

Given an image $I_{x}$, the target image $G(I_{x})$ can be obtained by the
generator, which consists of an Encoder and Decoder. Meanwhile, the
discriminator will determine whether the generated image is fake or real.
Based on the original GAN foundation, our proposed network extracts the
Encoder in the generator and uses it to extract the feature maps of the source
image $I_{x}$ and the target image $G(I_{x})$ respectively. At this point, due
to the dimensionality reduction operation of convolution, the feature map can
be seen as dividing the image into patches of the same size as the perceptual
field.

The original CUT_GAN randomly samples patches in the feature map to compute
the Contrastive loss. However, not all patches have rich domain information,
so random sampling causes the GAN network to not maximize the information
learned between two domains. To address this problem, we propose an additional
Attention Module to intentionally select more significant patches for
computing the Contrastive loss. The attention mechanism module calculates the
significance of each patch, and by ranking them we select patches that are
more important, i.e. contain more information about the domains.

Finally, all these sampled patches are used to compute the self-supervised
contrastive loss. This loss term encourages two elements (corresponding
patches) to map to a similar point in a learned feature space, relative to
other elements (patches) in the dataset, referred to as negatives. The output
can as results take on the appearance of the target domain while retaining the
structure, or content, of the specific input.

###  3.2 Generator

The generator G is designed to map the latent space vector (z) to data space.
Since our data are images, converting z to data space means ultimately
creating an RGB image with the same size as the training images (i.e.
3$\times$64$\times$64).

Empirically, the depth of the network is critical to the performance of the
model. When the number of layers is increased, the network can perform more
complex feature pattern extraction, so theoretically better results can be
achieved when the model is deeper. However, experiments have revealed a
degradation problem in deep networks: as the depth of the network increases,
the accuracy of the network saturates and even decreases.

To address this problem, we propose a ResNet-based generator G[11] consists of
two down-sampling blocks, nine intermediate blocks, two up-sampling blocks and
two convolution layers. We apply Instance Normalization (IN)[12][13] and
ReLU[14] in the generator, except for the output layer, which uses Tanh as the
activation function. The network before the sixth residual block is regarded
as the encoder E and the rest is the decoder. We adopt the multi-layer feature
extraction in CUT_GAN[15], which takes the features from five layers.
including the input image, the first and second down-sampling blocks, and the
first and fifth residual blocks.

###  3.3 Discriminator

The discriminator D is a binary classification network that takes an image as
input and outputs a scalar probability that the input image is real (as
opposed to fake).

The discriminator in our proposed network uses a design called PatchGAN[16].
In general, the discriminator of the original GAN is designed to output only
one boolean value (True or False), which is an evaluation of the whole image
generated by the generator. In contrast, PatchGAN is designed to be fully
convolutional, where the image is mapped using convolution into an $N\times N$
matrix, which is equivalent to the final evaluation value in the original GAN.
Each point (true or false) in the $N\times N$ matrix represents a small region
(in the sense of patch) of the original image, which is the application of the
perceptual field, which is shown in Figure 3. Instead of measuring the whole
image with a single value, the whole image is now evaluated using an $N\times
N$ matrix, which obviously allows more areas to be focused on, which is the
advantage of PatchGAN.

Fig. 3: Perceptual field

###  3.4 Patchwise Contrastive Learning

Image translation needs to ensure consistency of content. More precisely,
important elements of the image such as pedestrians, cars, traffic lights, and
even road patterns should be preserved as much as possible. Contrastive
learning can then solve this problem. Specifically, a patch is selected from
the real image as query; a number of patches are selected from the simulated
image, and the patches with the same position as query are used as positive
samples, while the others are used as negative samples.

Inspired by [17][18][15], we use PatchNCE[15] as contrastive loss. NCE, which
is called "Noise Contrastive Estimation", estimates model parameters and
normalization constants by maximizing the same objective function. NCE
converts the problem into a binary classification problem, where the
classifier is able to bifurcate the data samples and the noise samples. In
addition, the negative samples for contrastive learning are selected patches
within a single input image, rather than other images from the dataset.

Figure 4 depicts the structure of Patchwise Contrastive Loss. As mentioned
before, a generator is an encoder-decoder architecture, which passes the real
and simulated images through this encoder, and selects the feature maps output
from the L layers of the encoder to calculate the contrastive loss in the
feature space instead of the original image, as shown in

$L_{PatchNCE}(G,H,X)=E_{x\sim
X}\sum_{l=1}^{L}\sum_{s=1}^{S_{l}}l(\hat{z_{l}^{s}},z_{l}^{s},z_{l}^{S\setminus
s}).$ (1)

Note that in the above equation, L represents a total of L layers of encoder
output feature maps to calculate the contrastive loss, different spatial
locations in the feature maps of different layers correspond to different
image patches in the original image. The deeper the layer, the larger the
corresponding patch. $S_{l}$ represents that the feature maps of the current
layer have a total of $S_{l}$ spatial locations.
$\hat{z_{l}}=H_{l}(G_{enc}^{l}(G(x)))$(where H stands for MLP) indicates that
the generated image is passed through the encoder of the generator, and the
feature maps output at the l-th layer of the encoder are obtained by MLP for
$\hat{z_{l}}$. $z_{l}=H_{l}(G_{enc}^{l}(x))$ indicates that the real image is
input to the encoder of the generator, and the feature maps output at the l-th
layer of the encoder are obtained by MLP for $z_{l}$. $\hat{z_{l}^{s}}$ is one
spatial location of $\hat{z_{l}}$. $\hat{z_{l}^{s}}$ is query, $z_{l}^{s}$ is
positive samples, $z_{l}^{S\setminus s}$ is negative samples, which indicates
locations other than s in $z_{l}$.

Fig. 4: Patchwise Contrastive Loss

Each spatial location corresponds to the original map is a patch, and each
spatial location in the new feature maps obtained by MLP. H is a vector whose
dimension is the number of channels. Since query, positive samples, and
negative samples are all vectors, we can give the introduction of InfoNCE
formula in

$l(v,v^{+},v^{-})=log[\frac{exp(v\cdot v^{+}/\tau)}{exp(v\cdot
v^{+}/\tau)+\sum_{n=1}^{N}exp(v\cdot v_{n}^{-}/\tau)}].$ (2)

Before passing vectors into the above formulas, they need to be normalized to
prevent space collapse or expansion. Essentially it is actually a cross-
entropy loss function, which can close the distance between the query and the
positive samples in the feature space, while pushing away the distance between
the query and the negative samples in the feature space. It is actually
equivalent to an N+1 class classification problem (classifying query with a
class corresponding to one positive sample and a class corresponding to N
negative samples), as shown in Figure 4.

##  4 Experiment setting

In this section, we describe the details of Attention-based CUT_GAN
Implementation, including used Datasets, loss function and evaluation metrics.

###  4.1 Datasets

The experiments are based on the Cityscapes ,which contains a diverse set of
stereo video sequences recorded in street scenes from 50 different cities,
with high quality pixel-level annotations of 5000 frames in addition to a
larger set of 20000 weakly annotated frames. and Playing for Data dataset
[19], which consists of 24966 densely labelled frames split into 10 parts for
convenience. The class labels are compatible with the CamVid and CityScapes
datasets, so that it’s easy to use while the target is Cityscapes dataset.

###  4.2 Training

To show the effect of the proposed patch-based contrastive loss, we
intentionally match the architecture and hyperparameter settings of CycleGAN,
except the loss function. This includes the ResNet-based generator[11] with 9
residual blocks, PatchGAN discriminator[16], Least Square GAN loss[4], batch
size of 1, and Adam optimizer[20] with learning rate 0.002.

Our full model Attention-based CUT_GAN is trained up to 400 epochs, which is
the same as CUT_GAN settings. Our encoder $G_{enc}$ is the first half of the
CycleGAN generator[5]. In order to calculate our multi-layer, patch-based
contrastive loss, we extract features from 5 layers, which are RGB pixels, the
first and second downsampling convolution, and the first and the fifth
residual block. The layers we use correspond to receptive fields of sizes 1
$\times$ 1, 9 $\times$ 9, 15 $\times$ 15, 35 $\times$ 35, and 99 $\times$ 99\.
For each layer’s features, we sample 256 locations arrcording to their
significance, and apply 2-layer MLP to acquire 256-dim final features. We
normalize the vector by its L2 norm.

###  4.3 Loss function

The loss of Attention-based CUT_GAN comprises three main parts: least square
loss ${\cal L}_{GAN}$ [4], PatchNCE loss ${\cal L}_{PatchNCE}(G,H,X)$ from
domain X, PatchNCE loss ${\cal L}_{PatchNCE}(G,H,Y)$ from domain Y, which are
shown in

$\begin{split}{\cal L}_{GAN}(G,D,X,Y)+\lambda_{X}{\cal
L}_{PatchNCE}(G,H,X)+\\\ \lambda_{Y}{\cal L}_{PatchNCE}(G,H,Y).\end{split}$
(3)

We choose $\lambda_{X}=1$ when we jointly train with the identity loss
$\lambda_{Y}=1$, and choose a larger value $\lambda_{X}=10$ without the
identity loss ($\lambda_{Y}=0$) to compensate for the absence of the
regularizer. Our model is relatively simple compared to recent methods that
often use 5-10 losses and hyper-parameters.

###  4.4 Evaluation Metrics

Aimed at assessing visual quality and discovered correspondence, we mainly use
three metrics to evaluate the generated images: Frechet Inception Distance,
Inception Score and Sliced Wasserstein Distance.

####  4.4.1 Frechet Inception Distance

Frechet Inception Distance(FID) is a measure of similarity between two
datasets of images, which empirically estimates the distribution of real and
generated images in a deep network space and computes the divergence between
them. FID was shown to correlate well with human judgement of visual quality
and is most often used to evaluate the quality of samples of Generative
Adversarial Networks. The smaller the value of FID, the closer the generated
image is to the data distribution of the target domain.

####  4.4.2 Inception Score

Inception Score evaluates the quality of GAN-generated images in terms of both
clearness and diversity.

####  4.4.3 Sliced Wasserstein Distance

SWD can measure image distribution mismatches or imbalances without additional
labels. The idea is to first obtain the representation of the high-dimensional
probability distribution in one dimension by linear mapping, and then
calculate the wasserstein distance of the one-dimensional representation of
the two probability distributions.

##  5 Results and Discussion

In this section, we show the experimental results of four attention modules
separately: Triplet Attention, External Attention, Self-attention and
Bottleneck Attention. In addition, the baseline GAN such as CycleGAN and
CUT_GAN are also presented and compared.

###  5.1 Qualitative Comparision

| CUT_GAN | CycleGAN | Self-attention | External_attention | BAM | Triplet
---|---|---|---|---|---|---
FID | 55.48 | 52.87 | 51.41 | 52.68 | 51.40 | 50.55
IS | 2.42 | 2.44 | 2.45 | 2.32 | 2.42 | 2.31
SWD | 395.78 | 287.18 | 354.234 | 390.29 | 388.73 | 346.21
Table 1: Qualitative Comparision of baselines(CUT_GAN and CycleGAN) and
Attention-based CUT_GAN with four different attention modules

Table 1 shows ablation results for the effect of each attention module in our
Attention-based CUT_GAN. We summarize the Fréchet Inception Distance(FID),
Inception Score(IS) and Wasserstein Distance(SWD) results on Attention-based
CUT_GAN with four different attention modules. In addition, CUT_GAN and
CycleGAN are shown as baselines. For the evaluation metric FID(less is
better), Triplet Attention module is the best, which obtains 50.55. For the
evaluation metric IS(more is better), Self-Attention module is the best, which
achieves 2.45. For the evaluation metric SWD(less is better), CycleGAN is the
best, which reaches only 287.18.

###  5.2 Discussion

We use Fréchet Inception Distance(FID) and Sliced Wasserstein Distance (SWD)
to evaluate the quality of translated images. FID and SWD both measure the
distance between two distributions of real and generated images, and lower
indicate the generated image is similar to the real one. In addition, we use
Inception Score as evaluation metrics, which considers both image quality and
diversity.

According to attention mechanisms, there are four settings in our model,
Triplet Attention, External Attention, Self-attention and Bottleneck
Attention. For the metric of FID, the translated results of Triplet-Attention
based model are more realistic than other methods. For the metric of IS, the
translated results of Self-Attention based model performs the best than other
methods. However, for the metric of SWD, our proposed Attention-based CUT_GAN
not exceeds CycleGAN. Further, considering the three evaluation metrics
together, our Attention-based CUT_GAN model (no matter which attention
mechanism is applied) performs better than the original CUT_GAN model.
Moreover, our method does not add extra model parameters in both generator G
and discriminator D.

Compared to baseline methods, our Attention-based CUT_GAN model has the
ability to translate the domain-relevant features accurately. Further, the
position and structure of the objects in the generated image are also
maintained partially invariant. For example, there are a large number of trees
being generated in baseline models, however, this is not present in the
simulated image(source domain). This problem is mitigated in our proposed
model.

##  6 Conclusion

This paper focuses on solving the third step towards the photo-realistic
virtualization of road objects, namely the intelligent post-processing
algorithms. Firstly, this paper proposes a Generative adversarial network to
realize unpaired image-to-image translation and transformation between the
simulated domain and the real domain. Secondly, we propose an Attention
Mechanism Module for cross-domain contrastive learning in the task of Image-
to-Image Translation. Instead of randomly selecting the anchor, positive and
negative samples to compute the contrastive loss, we measure the significance
of source domain features and select them based on significance so that the
constraint becomes more relevant for the domain translation. Finally, we
experimentally prove that our Attention-based CUT_GAN outperforms the original
CUT_GAN and increases the mutual information between the source domain and
target domain.

For the Attention Mechanism Module, we apply four different attention
mechanisms: Triplet Attention, External Attention, Self-attention and
Bottleneck Attention. All settings outperform the original CUT_GAN without
adding extra model parameters in both generator and discriminator. Besides, we
process all generated images and convert them into training datasets of
semantic segmentation, in order to verify the practical value of this thesis.
After training, the semantic segmentation network can achieve higher accuracy
when tested with real images. However, it is still not good as the network is
trained with real images. We demonstrated the effectiveness of introducing the
attention mechanism module, but at the same time, there still has a lot of
work to be done to improve it.

## References

  * [1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial networks,”  Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.
  * [2] Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez,  “Invertible conditional gans for image editing,”  arXiv preprint arXiv:1611.06355, 2016.
  * [3] Alec Radford, Luke Metz, and Soumith Chintala,  “Unsupervised representation learning with deep convolutional generative adversarial networks,”  arXiv preprint arXiv:1511.06434, 2015.
  * [4] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley,  “Least squares generative adversarial networks,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2794–2802.
  * [5] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros,  “Unpaired image-to-image translation using cycle-consistent adversarial networks,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
  * [6] Andrew Brock, Jeff Donahue, and Karen Simonyan,  “Large scale gan training for high fidelity natural image synthesis,”  arXiv preprint arXiv:1809.11096, 2018.
  * [7] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros,  “Image-to-image translation with conditional adversarial networks,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [8] Augustus Odena, Christopher Olah, and Jonathon Shlens,  “Conditional image synthesis with auxiliary classifier gans,”  in International conference on machine learning. PMLR, 2017, pp. 2642–2651.
  * [9] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang,  “Self-supervised learning: Generative or contrastive,”  IEEE Transactions on Knowledge and Data Engineering, 2021.
  * [10] Minguk Kang and Jaesik Park,  “Contragan: Contrastive learning for conditional image generation,”  Advances in Neural Information Processing Systems, vol. 33, pp. 21357–21369, 2020.
  * [11] Justin Johnson, Alexandre Alahi, and Li Fei-Fei,  “Perceptual losses for real-time style transfer and super-resolution,”  in European conference on computer vision. Springer, 2016, pp. 694–711.
  * [12] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky,  “Instance normalization: The missing ingredient for fast stylization,”  arXiv preprint arXiv:1607.08022, 2016.
  * [13] Sergey Ioffe and Christian Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  in International conference on machine learning. PMLR, 2015, pp. 448–456.
  * [14] Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al.,  “Rectifier nonlinearities improve neural network acoustic models,”  in Proc. icml. Atlanta, Georgia, USA, 2013, vol. 30, p. 3.
  * [15] Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu,  “Contrastive learning for unpaired image-to-image translation,”  in European Conference on Computer Vision, 2020.
  * [16] Jinlin Ma, Meng Wei, Ziping Ma, Li Shi, and Kai Zhu,  “Retinal vessel segmentation based on generative adversarial network and dilated convolution,”  in 2019 14th International Conference on Computer Science & Education (ICCSE). IEEE, 2019, pp. 282–287.
  * [17] Michael Gutmann and Aapo Hyvärinen,  “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,”  in Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010, pp. 297–304.
  * [18] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel,  “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,”  Advances in neural information processing systems, vol. 29, 2016\.
  * [19] Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun,  “Playing for data: Ground truth from computer games,”  in European Conference on Computer Vision (ECCV), Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, Eds. 2016, vol. 9906 of LNCS, pp. 102–118, Springer International Publishing.
  * [20] P Kingma Diederik and Jimmy Ba Adam,  “A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.


# Towards One Shot Search Space Poisoning in Neural Architecture Search

Nayan Saxena1, Robert Wu , Rohan Jain

###### Abstract

We evaluate the robustness of a Neural Architecture Search (NAS) algorithm
known as Efficient NAS (ENAS) against data agnostic poisoning attacks on the
original search space with carefully designed ineffective operations. We
empirically demonstrate how our one shot search space poisoning approach
exploits design flaws in the ENAS controller to degrade predictive performance
on classification tasks. With just two poisoning operations injected into the
search space, we inflate prediction error rates for child networks upto 90% on
the CIFAR-10 dataset.

## Introduction

The problem of finding optimal deep learning architectures has recently been
automated by neural architecture search (NAS) algorithms. These algorithms
continually sample operations from a predefined search space to construct
neural networks to optimize a performance metric over time, eventually
converging to better child architectures. This intuitive idea greatly reduces
human intervention by restricting human bias in architecture engineering to
just the selection of the predefined search space (Elsken et al. 2019).
Although NAS has the potential to revolutionize architecture search across
many applications, human selection of the search space remains a security risk
that needs to be evaluated before NAS can be deployed in security-critical
domains. While NAS has been studied to further develop more adversarially
robust networks through addition of dense connections (Guo et al. 2020),
little work has been done in the past to assess the adversarial robustness of
NAS itself. Search phase analysis has shown that computationally efficient
algorithms such as ENAS are worse at truly ranking child networks due to their
reliance on weight sharing (Yu et al. 2019). Finally, most traditional
poisoning attacks involve injecting mislabeled examples in the training data
and have been executed against classical machine learning approaches
(Schwarzschild et al. 2021). We validate these concerns by evaluating the
robustness of one such NAS algorithm known as Efficient NAS (ENAS) (Pham et
al. 2018) against data-agnostic search space poisoning (SSP) attacks on the
CIFAR-10 dataset. Throughout this paper, we focus on the pre-optimized ENAS
search space $\hat{\mathcal{S}}$ = {Identity, 3x3 Separable Convolution, 5x5
Separable Convolution, Max Pooling (3x3), Average Pooling (3x3)} (Pham et al.
2018).

## Search Space Poisoning (SSP)

The idea behind SSP, as shown in Figure 1, is to inject precisely designed
multiset $\mathcal{P}$ of ineffective operations into the ENAS search space,
making the search space $\mathcal{S}:=\hat{\mathcal{S}}\cup\mathcal{P}$. Our
approach exploits the core functionality of the ENAS controller to sample
child networks from a large computational graph of operations by introducing
highly ineffective local operations into the search space. On the attacker’s
behalf, this requires no a priori knowledge of the problem domain or dataset
being used, making this new approach more favourable than traditional data
poisoning attacks.

Figure 1: Overview of Search Space Poisoning (SSP) Figure 2: Final validation
and test classification errors as a function of multiple operation instances.
(a) Identity layers were moderately effective (b) Gaussian noise reached high
error rates even with fewer operations (c) Dropout proved most effective (d)
Transposed convolutions plateaued after a saturation point.

### Multiple-Instance Poisoning

As a naïve strategy, we first propose multiple-instance poisoning which
increases the likelihood of sampling bad operations by including duplicates of
these bad operations in the search spaces. Through experimental results we
discovered that biasing the search space this way resulted in final networks
that are mostly comprised of these poor operations with error rates exceeding
80%. However, as shown in Figure 2, to perform well this approach requires
overwhelming the original search space with up to 300 bad operations (50:1
ratio of bad operations per each good operation) which is unreasonable. The
motivation then is to reduce the ratio of bad to good operations down to 1:1,
or even lower, to make search space poisoning more feasible and effective.

## Towards One Shot Poisoning

In an attempt to improve the attack, we further attempted to reduce the number
of poisoning points to just 2 points by adding: (i)
$\textnormal{Dropout}(p=1)$ (ii) Stretched
$\textnormal{Conv}(k=3,\textnormal{padding, dilation}=50)$ to the original
search space. Our rationale is that dropout operations with $p=1$ would erase
all information and produce catastrophic values such as $0$ or not-a-number
(NaN). The results were promising, with error rates shooting up to 90% very
quickly during training as seen in Figure 3 and Table 1. An example final
child network producing these high errors can be observed in Figure 4.

Figure 3: (a) Validation error for one shot poisoning over $300$ epochs (b)
Distribution of bad operations sampled by the ENAS controller after 300
epochs.

Search Space | $|\mathcal{P}|$ | Val Error | Test Error
---|---|---|---
$\hat{\mathcal{S}}\textnormal{ (Baseline)}$ | 0 | $16.4\%$ | $19.8\%$
$\hat{\mathcal{S}}+300\\{\textnormal{Dropout}(p=1)\\}$ | 300 | ${84.8\%}$ | ${84.3\%}$
$\hat{\mathcal{S}}+\\{{\textnormal{Conv}(k=3,p,d=50),\textnormal{ Dropout}(p=1)}\\}$ | 2 | 90.1% | 90.0%

Table 1: Experimental results showing how one shot poisoning proves
surprisingly effective with just 2 points as compared to its multiple instance
counterpart with 300 points.

InputSoftmaxConv 3x3Sep 5x5StretchedConv 3x3Conv 3x3Dropout(p=1)Avg.
PoolStretchedSep 3x3Conv 3x3Max PoolConv 3x3

Figure 4: Network produced by ENAS after one shot poisoning with good
operations highlighted in green and poisoning operations highlighted in red.
Search space utilized is the same as shown in Table 1 with two poisoning
points.

## Conclusion

In this paper, we focused on examining the robustness of ENAS under our newly
proposed SSP paradigm. Our carefully designed poisoning sets demonstrated the
potential to make it incredibly easy for an attacker with no prior knowledge
or access to the training data to still drastically impact the quality of
child networks. Finally, our one-shot poisoning results reveal an opportunity
for future work in neural architecture design, as well as challenges to
surmount in using NAS for more adversarially robust architecture.

## Acknowledgements

The authors would like to thank Chuan-Yung Tsai & George-Alexandru Adam for
their valuable comments. We thank the ML Collective community for the generous
computational support and feedback on this research. We are also grateful to
Kanav Singla and Benjamin Zhuo for their contributions to the codebase.

## References

  * Elsken et al. (2019) Elsken, T.; Metzen, J. H.; Hutter, F.; et al. 2019.  Neural Architecture Search: A Survey.  _J. Mach. Learn. Res._ , 20(55): 1–21.
  * Guo et al. (2020) Guo, M.; Yang, Y.; Xu, R.; Liu, Z.; and Lin, D. 2020.  When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 631–640.
  * Pham et al. (2018) Pham, H.; Guan, M.; Zoph, B.; Le, Q.; and Dean, J. 2018.  Efficient Neural Architecture Search via Parameters Sharing.  In _International Conference on Machine Learning_ , 4095–4104. PMLR.
  * Schwarzschild et al. (2021) Schwarzschild, A.; Goldblum, M.; Gupta, A.; Dickerson, J. P.; and Goldstein, T. 2021\.  Just how toxic is data poisoning? a unified benchmark for backdoor and data poisoning attacks.  In _International Conference on Machine Learning_ , 9389–9398. PMLR.
  * Srivastava et al. (2014) Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1): 1929–1958.
  * Yu et al. (2019) Yu, K.; Sciuto, C.; Jaggi, M.; Musat, C.; and Salzmann, M. 2019.  Evaluating the Search Phase of Neural Architecture Search.  _arXiv preprint arXiv:1902.08142_.

Supplementary Material

## Search Space Poisoning (SSP)

Poisoning Set | Search Space | Experiment | Poisoning Multiset | Validation Error | Test Error
---|---|---|---|---|---
$\mathcal{P}$ | $\mathcal{S}$ |  | $q(\mathcal{P})$ |  |
$\varnothing$ | $\hat{\mathcal{S}}$ | Original | $\varnothing$ | $19.53$ | $25.33$
|  | 1a | $6(\mathcal{P}_{1})$ | $24.40$ | $30.93$
$\mathcal{P}_{1}=$ {Identity} | $\mathcal{S}_{1}=\hat{\mathcal{S}}\cup\mathcal{P}_{1}$ | 1b | $36(\mathcal{P}_{1})$ | $38.54$ | $41.31$
|  | 1c | $120(\mathcal{P}_{1})$ | $50.07$ | $52.38$
|  | 1d | $300(\mathcal{P}_{1})$ | $\mathbf{69.78}$ | $\mathbf{67.97}$
|  | 2a | $6(\mathcal{P}_{2})$ | $43.71$ | $46.73$
$\mathcal{P}_{2}=$ {Gaussian ($\sigma=10$)} | $\mathcal{S}_{2}=\hat{\mathcal{S}}\cup\mathcal{P}_{2}$ | 2b | $36(\mathcal{P}_{2})$ | $73.64$ | $73.82$
|  | 2c | $120(\mathcal{P}_{2})$ | $84.94$ | $84.44$
|  | 2d | $300(\mathcal{P}_{2})$ | $\mathbf{86.26}$ | $\mathbf{85.49}$
|  | 3a | $6(\mathcal{P}_{3})$ | $32.23$ | $37.60$
$\mathcal{P}_{3}=$ {Dropout ($p=1.0$)} | $\mathcal{S}_{3}=\hat{\mathcal{S}}\cup\mathcal{P}_{3}$ | 3b | $36(\mathcal{P}_{3})$ | $48.67$ | $51.88$
|  | 3c | $120(\mathcal{P}_{3})$ | $70.63$ | $71.16$
|  | 3d | $300(\mathcal{P}_{3})$ | $\mathbf{84.89}$ | $\mathbf{84.31}$
|  | 4a | $3(\mathcal{P}_{4})$ | $19.50$ | $22.43$
$\mathcal{P}_{4}=$ {3x3 transposed convolution, | $\mathcal{S}_{4}=\hat{\mathcal{S}}\cup\mathcal{P}_{4}$ | 4b | $18(\mathcal{P}_{4})$ | $36.00$ | $37.15$
5x5 transposed convolution} |  | 4c | $60(\mathcal{P}_{4})$ | $67.53$ | $63.89$
|  | 4d | $150(\mathcal{P}_{4})$ | $\mathbf{67.68}$ | $\mathbf{64.14}$
|  | 5a | $1(\mathcal{P}_{2}+\mathcal{P}_{5})$ | $29.38$ | $33.73$
$\mathcal{P}_{5}:=\mathcal{P}_{1}\cup\mathcal{P}_{2}\cup\mathcal{P}_{3}\cup\mathcal{P}_{4}$ | $\mathcal{S}_{5}=\hat{\mathcal{S}}\cup\mathcal{P}_{5}$ | 5b | $6(\mathcal{P}_{2}+\mathcal{P}_{5})$ | $55.06$ | $56.02$
|  | 5c | $20(\mathcal{P}_{2}+\mathcal{P}_{5})$ | $\mathbf{80.88}$ | $\mathbf{79.03}$
|  | 5d | $50(\mathcal{P}_{2}+\mathcal{P}_{5})$ | $72.56$ | $70.14$

Table 2: Summary of experimental search spaces with corresponding final
validation and test accuracies for SSP. Note that the multiset seed for
experiments 5a-5d includes two instances of $P_{2}$ to convenient round out
the cardinality of the multisets.

### Multiple-Instance Poisoning Attacks

Over the course of training, the LSTM controller paired with the RL search
strategy in ENAS develops the ability to sample architectures with operations
that most optimally reduce the validation error. We propose multiple-instance
poisoning which essentially increases the likelihood of a poisonous operation
$o_{\scaleto{\mathcal{P}}{3.4pt}}$ being sampled from the poisoned search
space $\mathcal{S}$. This is achieved by increasing the frequency of sampling
$o_{\scaleto{\mathcal{P}}{3.4pt}}$ from $\mathcal{S}$ through the inclusion of
multiple-instances of each $o_{\scaleto{\mathcal{P}}{3.4pt}}$ from the
poisoning multiset, so-called to allow for duplicate elements. An instance
factor $q\in\mathbb{N}^{\geq 1}$ represents instance multiplication of
$o_{\scaleto{\mathcal{P}}{3.4pt}}$ in the multiset $q$ times. Henceforth, the
probabilities of sampling
$o_{\scaleto{\hat{\mathcal{{S}}}}{5pt}}\in\hat{\mathcal{{S}}}$ and
$o_{\scaleto{\mathcal{P}}{3.4pt}}\in\mathcal{P}$, respectively, are,
${Pr}[o_{\scaleto{\hat{\mathcal{S}}}{5pt}}]:=\frac{1}{|\mathcal{S}|+q|\mathcal{P}|}$
and $\
{Pr}[o_{\scaleto{\mathcal{P}}{3.4pt}}]:=\frac{q}{|\mathcal{S}|+q|\mathcal{P}|}$.
From which, it is evident that under a multiple-instance poisoning framework,
the probability of sampling poisoned operations is strictly greater than
sampling operations in $\hat{\mathcal{S}}$; that is,
${Pr}[o_{\scaleto{\hat{\mathcal{S}}}{5pt}}]<{Pr}[o_{\scaleto{\mathcal{P}}{3.4pt}}]$.
It is also important to note that our technique is not the typical image-
agnostic perturbation/universal adversarial perturbation
$\delta\in\mathbb{R}^{d}$ intended to fool the target neural network $f$ on
almost all the input images from the target distribution $X$. Finally, another
challenge to overcome within our search space poisoning framework is to craft
each $o_{\scaleto{\mathcal{P}}{3.4pt}}\in\mathcal{P}$ such that it counteracts
the efficacy of the original operations
$o_{\scaleto{\hat{\mathcal{S}}}{5pt}}\in\hat{\mathcal{S}}$, which we tackle in
the next section.

### Crafting Poisoning Sets with Operations

##### Identity Operation

The simplest way to attack the functionality of ENAS is to inject non-
operations within the original search space which keep the input and outputs
intact. As a result, the controller will sample child models with layers
representing computations which preserve the inputs, making the operation
highly ineffective within a network architecture. This goal is achieved by the
inexpensive identity operation which has no numerical effect on the inputs. It
should also be noted that, the identity operation is not a skip connection.
Therefore, we define our first poisoning set: $\mathcal{P}_{1}:=$ {Identity}.

##### Gaussian Noise Layer

Typically used in signal processing, electronics and mitigating over-fitting
as some form of random data augmentation. Gaussian noise is a type of
statistical noise which is in the form of a Normal distribution
($X\sim\mathcal{N}(\mu,\sigma^{2})$). In PyTorch/Keras, the Gaussian noise
layer additively applies zero-centered Gaussian noise passing in the argument
of relative standard deviation used to generate the noise. We hypothesize that
including such layers with increasingly varied relative standard deviations
such as $\sigma=10$, can significantly impact the accuracy of the generated
child models making our poisoning set $\mathcal{P}_{2}:=$ {Gaussian
($\sigma=10$)}.

##### Dropout Layer

While dropout layers have historically been shown to be useful in preventing
neural networks from over-fitting (Srivastava et al. 2014), a high dropout
rate can result in severe information loss leading to poor performance of the
overall network. This is because given a dropout probability
$p\in\mathbb{[}0,1]$, dropout randomly zeroes out some values from the input
to decorrelate neurons during training. We hypothesize that including such
layers with high dropout probability, such as $p=1$, has the potential to
contaminate the search space with irreversible effects on the training
dynamics of ENAS. Here, we define our poisoning set as $\mathcal{P}_{3}:=$
{Dropout ($p=1$)}.

##### Transposed Convolutions

As described earlier, amongst other useful operations the original ENAS search
space $\hat{\mathcal{S}}$ also contains 3x3 and 5x5 convolutional layers
(separable). Intuitively, transposed convolutions upsample the input feature
map. It is important to note that transposed convolutions do not perform like
deconvolutional layers; they actually swap the forward and backward passes of
a convolution. Transposed convolutions, also known as fractionally strided
convolutions, stride over the output which is equivalent to a fractional
stride over the input. We define our poisoning set: $\mathcal{P}_{4}=$ {3x3
transposed convolution, 5x5 transposed convolution}.

## Experiments

#### Experimental Setup

To test the effectiveness of our proposed approach, we designed experiments
based on previously described methods outlined in Table 1. Each experiment
involved training ENAS on the CIFAR-10 dataset for 300 epochs. The results
presented in this paper are the average of three runs per experiment.

Hyperparameter | Value
---|---
search_for | macro
dataset | CIFAR10 or CIFAR100
n_classes | 10 or 100
n_train | 45000
n_val | 5000
batch_size | $128$
search_for | $300$
seed | $69$
cutout | $0$
fixed_arc | False
child_num_layers | $12$
child_out_filters | $36$
child_grad_bound | $5.0$
child_l2_reg | $0.00025$
child_keep_prob | $0.9$
child_lr_max | $0.05$
child_lr_min | $0.0005$
child_lr_T | $10$
controller_lstm_size | $64$
controller_lstm_num_layers | $1$
controller_entropy_weight | $0.0001$
controller_train_every | $1$
controller_num_aggregate | $20$
controller_train_steps | $50$
controller_lr | $0.001$
controller_tanh_constant | $1.5$
controller_op_tanh_reduce | $2.5$
controller_skip_target | $0.4$
controller_skip_weight | $0.8$
controller_bl_dec | $0.99$
p (Dropout Rate) | $1.0$
Table 3: Summary of experiment hyperparameters

For a baseline, we created an experiment with the original search space
$\hat{\mathcal{S}}$ (which was run three times). Because we wanted to evaluate
the poisoning effectiveness of each operation independently, we created a
group of four experiments for each poisoning set $P_{i}$. We henceforth refer
to these experiments as “Space $i$a” through “Space $i$d”. Following multiple-
instance poisoning attacks, each experiment includes a poisoning multiset
$q(P_{i})$ where $|q(P_{i})|=6,36,120,300$ for Space $i$a, Space $i$b, Space
$i$c, Space $i$d, respectively. Of note is that the instance-factor $q$ is
dependent on the experiment index and $|P_{i}|$. For example: Space 1c/2c/3c
has $|P_{1}|=|P_{2}|=|P_{3}|=1$, and $q|P_{1}|=q|P_{2}|=q|P_{3}|=120$, so
$q=120$; Space 4c has $|P_{4}|=2$ and $q|P_{4}|=120$, so $q=60$; Space 5c/6c
have $|P_{5}|=|P_{6}|=6$ and $q|P_{5}|=q|P_{6}|=120$, so $q=20$.

The software used includes Python (3.6.x-3.8.x) and PyTorch (1.9), with CUDA
(10.2, 11.1). Our hardware varies: CPUs included 2nd gen Xeon E, 3rd gen Core
i5, 8th gen Core i3; GPUs included Nvidia GTX {1050, 1050 Ti, 1080}, RTX 2060,
Tesla {K80, P100 (Google Colaboratory)}, TITAN Xp. Each experiment was
processed with only one GPU with 3GB of allocated VRAM, and took between 18-36
real-world hours.

### Experimental Results for One-Shot Poisoning

Poisoning Set | Search Space | Cardinality | Val Error | Test Error
---|---|---|---|---
$\mathcal{P}_{i}^{+}$ | $|\mathcal{P}_{i}^{+}|$ | $\mathcal{S}_{i}^{+}$ |  |
$\mathcal{P}_{0}^{+}=$ {Stretched Conv $(k=3,p,d=50)$} | $\mathcal{S}_{6}^{+}=\hat{\mathcal{S}}+6(\mathcal{P}_{0}^{+})$ | $6$ | $39.49\%$ | $42.87\%$
$\mathcal{P}_{2}^{+}=\mathcal{P}_{2}+\mathcal{P}_{0}^{+}$ | $6$ | $\mathcal{S}_{2}^{+}=\hat{\mathcal{S}}+6(\mathcal{P}_{2}^{+})$ | $35.92\%$ | $40.35\%$
$\mathcal{P}_{3}^{+}=\mathcal{P}_{3}+\mathcal{P}_{0}$ | $\mathbf{2}$ | $\mathcal{S}_{3}^{+}=\hat{\mathcal{S}}+2(\mathcal{P}_{3}^{+})$ | $\mathbf{90.12}\%$ | $\mathbf{90.00\%}$
$\mathcal{P}_{4}^{+}=\mathcal{P}_{4}+\mathcal{P}_{0}$ | $6$ | $\mathcal{S}_{4}^{+}=\hat{\mathcal{S}}+6(\mathcal{P}_{4}^{+})$ | $29.75\%$ | $36.46\%$
$\mathcal{P}_{5}^{+}=\mathcal{P}_{5}+\mathcal{P}_{0}$ | $6$ | $\mathcal{S}_{5}^{+}=\hat{\mathcal{S}}+6(\mathcal{P}_{5}^{+})$ | $28.27\%$ | $31.79\%$
$\mathcal{P}_{3}^{+,2}=\mathcal{P}_{3}+\mathcal{P}_{0}$ | $\mathbf{2}$ | $\mathcal{S}_{3}^{+}=\hat{\mathcal{S}}+2(\mathcal{P}_{3}^{+,2})$ | $\mathbf{90.12}\%$ | $\mathbf{90.00\%}$

Table 4: Summary of experimental search spaces with corresponding final
validation and test errors for one shot SSP. Note the humble cardinality of
$\mathcal{P}_{3}^{+}$ achieving $90\%+$ error.

### Our Files

  * •

The files in which these extended search spaces can be found are in
enas_poisoning/.

  * •

Initial experimental results can be found in ENAS-Experiments/results.


# Feature-Augmented Neural Networks for Patient Note De-identification

Ji Young Lee1∗, Franck Dernoncourt1∗, Özlem Uzuner2, Peter Szolovits1
1MIT, 2SUNY Albany
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
∗ These authors contributed equally to this work

###### Abstract

Patient notes contain a wealth of information of potentially great interest to
medical investigators. However, to protect patients’ privacy, Protected Health
Information (PHI) must be removed from the patient notes before they can be
legally released, a process known as patient note de-identification. The main
objective for a de-identification system is to have the highest possible
recall. Recently, the first neural-network-based de-identification system has
been proposed, yielding state-of-the-art results. Unlike other systems, it
does not rely on human-engineered features, which allows it to be quickly
deployed, but does not leverage knowledge from human experts or from
electronic health records (EHRs). In this work, we explore a method to
incorporate human-engineered features as well as features derived from EHRs to
a neural-network-based de-identification system. Our results show that the
addition of features, especially the EHR-derived features, further improves
the state-of-the-art in patient note de-identification, including for some of
the most sensitive PHI types such as patient names. Since in a real-life
setting patient notes typically come with EHRs, we recommend developers of de-
identification systems to leverage the information EHRs contain.

00footnotetext:  This work is licenced under a Creative Commons Attribution
4.0 International License.
License details: http://creativecommons.org/licenses/by/4.0/

##  1 Introduction and related work

Medical practitioners increasingly store patient data in Electronic Health
Records (EHRs) [Hsiao et al., 2011], which represents a considerable
opportunity for medical investigators to construct novel models and
experiments to improve patient care. Some governments even subsidize the
adoption of EHRs, such as the Centers for Medicare & Medicaid Services in the
United States who have spent over $30 billion in EHR incentive payments to
hospitals and medical providers [McCann, 2015].

A legal prerequisite for a patient note to be shared with a medical
investigator is that it must be de-identified. The objective of the de-
identification process is to remove all Protected Health Information (PHI).
Not appropriately removing PHI may result in financial penalties [DesRoches et
al., 2013, Wright et al., 2013]. In the United States, the Health Insurance
Portability and Accountability Act (HIPAA) [Office for Civil Rights, 2002]
defines PHI types that must be removed, ranging from phone numbers to patient
names. Failure to accurately de-identify a patient note would jeopardize the
patient’s privacy: the performance of a de-identification system is therefore
critical.

A naive approach to de-identification is to manually identify PHI. However,
this is costly [Douglass et al., 2005, Douglas et al., 2004] and unreliable
[Neamatullah et al., 2008]. Consequently, there has been much work developing
automated de-identification systems. These systems are either based on rules
or machine-learning models. Rule-based systems typically rely on patterns,
expressed as regular expressions and gazetteers, defined and tuned by humans
[Berman, 2003, Beckwith et al., 2006, Fielstein et al., 2004, Friedlin and
McDonald, 2008, Gupta et al., 2004, Morrison et al., 2009, Neamatullah et al.,
2008, Ruch et al., 2000, Sweeney, 1996, Thomas et al., 2002].

Machine-learning-based systems train a classifier to label each token as PHI
or not PHI. Some systems are more fine-grained by detecting which PHI type a
token belongs to. Different statistical methods have been explored for patient
note de-identification, including decision trees [Szarvas et al., 2006], log-
linear models, support vector machines (SVMs) [Guo et al., 2006, Uzuner et
al., 2008, Hara, 2006], and conditional random field (CRFs) [Aberdeen et al.,
2010]. A thorough review of existing systems can be found in [Meystre et al.,
2010, Stubbs et al., 2015].

A more recent system has introduced the use of artificial neural networks
(ANNs) for de-identification [Dernoncourt et al., 2016], and obtained state-
of-the-art results. The system does not use any manually-curated features.
Instead, it solely relies on character and token embeddings. While this allows
the system to be developed and deployed faster, it fails to give users the
possibility to add features engineered by human experts. Additionally, in
practical settings of de-identification, patient notes typically come from a
hospital EHR database, which contains metadata such as which patient each note
pertains to, and other information such as the names of all doctors who work
at the hospital where the patient was treated. The features derived from EHR
databases may be useful for boosting the performance of de-identification
systems. In this work, we present a method to incorporate features to this
ANN-based system, and show that it further improves the state-of-the-art.

##  2 Method

The first model based on ANNs for patient note de-identification was
introduced in [Dernoncourt et al., 2016]: we extend upon their model. They
utilized both token and character embeddings to learn effective features from
data by fine-tuning the parameters. The main components of the ANN model are
Long Short Term Memories (LSTMs) [Hochreiter and Schmidhuber, 1997], which are
a type of recurrent neural networks (RNNs).

The model is composed of three layers: a character-enhanced token embedding
layer, a label prediction layer, and a label sequence optimization layer. The
character-enhanced token embedding layer maps each token into a vector
representation. The sequence of vector representations corresponding to a
sequence of tokens are input to the label prediction layer, which outputs the
sequence of vectors containing the probability of each label for each
corresponding token. Lastly, the sequence optimization layer outputs the most
likely sequence of predicted labels based on the sequence of probability
vectors from the previous layer. All layers are learned jointly. For more
details on the basic ANN model, see [Dernoncourt et al., 2016].

We augment this ANN model by adding features that are human-engineered or
derived from EHR database, as presented in Table 1. The majority of human-
engineered features are taken from [Filannino and Nenadic, 2015], a few more
features come from [Yang and Garibaldi, 2015], and additional gazetteers are
collected using online resources. All features are binary and computed for
each token. The binary feature vector comprising all features for a given
token is fed into a feedforward neural network, the output vector of which is
concatenated to the corresponding token embeddings, at the output of the
character-enhanced token embedding layer, as Figure 1 illustrates.

Figure 1: Feature-augmented token embeddings. Each token is mapped to a token embedding that is the concatenation of three elements: the output of a feedforward neural network that takes the features as input, a pre-trained token embedding, and the output of a bidirectional-LSTM (bi-LSTM) that takes the character embeddings as input.  Feature types |  Features
---|---
| Note metadata
---
Hospital data
$\left.\begin{tabular}[]{@{\ }l@{}}\text{Patient's first name, patient's last
name}\\\ \text{Doctor's first names, doctor's last
names}\end{tabular}\right\\}$ EHR features
Morphological |  Ends with s, is the first letter capitalized, contains a digit, is numeric, is alphabetic, is alphanumeric, is title case, is all lower case, is all upper case, is a stop word
Semantic/Wordnet |  Hypernyms, senses, lemma names
Temporal |  Seasons, months, weekdays, times of the day, years, years followed by apostrophe, festivity dates, holidays, cardinal numbers, decades, fuzzy quantifier (e.g., “approximately”, “few”), future trigger (e.g., “next”, “tomorrow”)
Gazetteers |  Honorifics for doctors, honorifics, medical specialists, medical specialties, first names, last names, last name prefixes, street suffixes, US cities, US states (including abbreviations), countries, nationalities, organizations, professions
Regular expressions |  Email, age, date, phone, zip code, id number, medical record number
Table 1: Feature list. Note metadata and hospital data are derived from the
EHR database. Morphological, semantic/wordnet, and temporal features are
commonly used features for NLP tasks. Gazetteers and regular expressions are
specifically engineered for the task.

##  3 Experiments

We evaluate our model on the de-identification dataset introduced in
[Dernoncourt et al., 2016], which is a subset of the MIMIC-III dataset
[Goldberger et al., 2000, Saeed et al., 2011, Johnson et al., 2016], using the
same train/validation/test split (70%/10%/20%). We chose this dataset as each
note comes with metadata, such as the patient’s name, and it is the largest
de-identification dataset available to us. It contains 1,635 discharge
summaries, 2,945,228 tokens, 69,525 unique tokens, and 78,633 PHI tokens.

The model is trained using stochastic gradient descent, updating all
parameters, i.e., token embeddings, character embeddings, parameters of
bidirectional LSTMs, and transition probabilities, at each gradient step. For
regularization, dropout is applied to the character-enhanced token embeddings
before the label prediction layer. We set the character embedding dimension to
25, the character-based token embedding LSTM dimension to 25, the token
embedding dimension to 100, the label prediction LSTM dimension to 100, the
dropout probability to 0.5, and we use GloVe embeddings [Pennington et al.,
2014] trained on Wikipedia and Gigaword 5 [Parker et al., 2011] articles as
pre-trained token embeddings. The hyperparameters were optimized based on the
performance on the validation set.

##  4 Results

Table 2 presents the main results. The epochs for which the results are
reported are optimized based on either the highest F1-score or the highest
recall on the validation set. As expected, choosing the epoch based on the
recall improves the recall on the test set, while lowering the precision.
Overall, adding features consistently improves the results.

Table 2: Binary HIPAA token-based results (%) for the ANN model, averaged over 5 runs. The metric refers to the detection of PHI tokens versus non-PHI tokens, amongst PHI types that are defined by HIPAA. “No feature” is the model utilizing only character and word embeddings, without any feature. “EHR features” uses only 4 features derived from EHR database: patient first name, patient last name, doctor first name, and doctor last name. “All features” makes use of all features, including the EHR features as well as other engineered features listed in Table 1. “Optimized by F1-score” and “optimized by recall” means that the epochs for which the results are reported are optimized based on the highest F1-score or the highest recall on the validation set, respectively. | Binary HIPAA (optimized by F1-score) | Binary HIPAA (optimized by recall)
---|---|---
Precision |  Recall |  F1-score |  Precision |  Recall |  F1-score
No feature |  99.103 |  99.197 |  99.150 |  98.557 |  99.376 |  98.965
EHR features |  99.100 |  99.304 |  99.202 |  98.771 |  99.441 |  99.105
All features |  99.213 |  99.306 |  99.259 |  98.880 |  99.420 |  99.149
Table 3: Binary token-based results (%) . The reported results are optimized by recall, and averaged over 5 runs. The symbol 1 indicates that the PHI type is not required by HIPAA. The PHI type “location” designates any location that is not a street name, zip code, state or country. P stands for precision, R for recall, and F1 for F1-score.  | No feature | EHR features | All features |
---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | Support
Zip | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 24
Date | 98.90 | 99.77 | 99.33 | 98.95 | 99.79 | 99.36 | 98.99 | 99.69 | 99.34 | 20627
Phone | 98.31 | 99.58 | 98.94 | 98.98 | 99.46 | 99.22 | 99.42 | 99.32 | 99.37 | 1438
Patient | 96.89 | 98.34 | 97.61 | 98.62 | 99.14 | 98.88 | 99.21 | 99.27 | 99.24 | 302
ID | 99.57 | 98.24 | 98.90 | 99.31 | 98.82 | 99.07 | 99.77 | 97.97 | 98.86 | 612
Doctor1 | 97.47 | 98.17 | 97.82 | 97.27 | 98.48 | 97.87 | 97.56 | 98.20 | 97.88 | 3676
Location | 96.02 | 95.71 | 95.86 | 96.41 | 96.49 | 96.45 | 96.65 | 96.32 | 96.46 | 462
Age $\geq$ 90 | 75.12 | 94.29 | 83.60 | 77.04 | 95.72 | 85.35 | 78.93 | 93.57 | 84.80 | 28
Hospital1 | 94.78 | 95.39 | 95.08 | 94.77 | 95.52 | 95.14 | 95.53 | 95.50 | 95.51 | 1259
State1 | 99.36 | 94.33 | 96.76 | 99.68 | 94.03 | 96.73 | 99.39 | 91.94 | 95.49 | 67
Street | 96.77 | 85.25 | 90.54 | 97.63 | 85.25 | 90.96 | 93.91 | 86.56 | 89.81 | 61
Country1 | 87.51 | 85.00 | 86.11 | 89.29 | 82.50 | 85.67 | 86.87 | 95.00 | 90.56 | 16
Binary | 98.41 | 99.19 | 98.80 | 98.48 | 99.27 | 98.87 | 98.61 | 99.15 | 98.88 | 28572

Table 3 details the results for each PHI type. The system using only the EHR
features yields the highest recall for 6 out of 12 PHI types. Most
importantly, the recall for patient and doctor names are higher when using
features than when using no feature: this is expected as the patient name of
the note and the doctor names are used as features. In fact, the two remaining
false negatives for patient names are annotation errors. For example, in the
sentence “The patient responded well to _Natrecor_ in the past, but the
improvement disappeared soon”, the drug name _Natrecor_ was incorrectly marked
as a patient name by the human annotator. This result is highly remarkable as
patient names are the most sensitive information in a patient note [South et
al., 2014].

Adding all features often lowers the recall compared to using EHR features
only, although the F1-score remains virtually unchanged. This is somewhat
surprising, as we had expected that the features would help, as using the same
feature set with a CRF to perform de-identification yields state-of-the-art
results next to the ANN models [Dernoncourt et al., 2016]. This could be
explained as follows. Human-engineered features tend to have higher precision
than recall, as it is often hard to design regular expressions or gazetteers
that can detect all possible instances or variations of the desired entities.
We conjecture that as the ANN model learn to rely more on such features, it
might lose the ability to learn to pick up tokens that deviate from engineered
features, resulting in a lower recall. For example, we notice that the phone
PHI tokens that are not detected by the model using all features but are
detected by the other two models, are ill-formed phone numbers such as
“617-554-$\mid$2395”, or phone extensions such as “617-690-4031 ext 6599”.
Since the phone regular expressions do not capture these two examples, they
are more likely to be false negatives in the model that uses the phone regular
expression features.

##  5 Conclusion

In this paper we presented an extension of the ANN-based model for patient
note de-identification that can incorporate features. We showed that adding
features results in an increase of the recall, in particular features
leveraging information from the associated EHRs, namely patient names and
doctor names.

Our results suggest that constructing patient note de-identification systems
should be performed using structured information from the EHRs, the latter
being available in a typical, real-life setting. We restricted our EHR-derived
features to patient and doctor names, but it could be extended to the many
other structured fields that EHR contain, such as patients’ addresses, phone
numbers, email addresses, professions, and ages.

## Acknowledgements

The project was supported by Philips Research. The content is solely the
responsibility of the authors and does not necessarily represent the official
views of Philips Research. We warmly thank Michele Filannino, Alistair
Johnson, Li-wei Lehman, Roger Mark, and Tom Pollard for their helpful
suggestions and technical assistance.

## References

  * [Aberdeen et al., 2010]  John Aberdeen, Samuel Bayer, Reyyan Yeniterzi, Ben Wellner, Cheryl Clark, David Hanauer, Bradley Malin, and Lynette Hirschman.  2010\.  The MITRE Identification Scrubber Toolkit: design, training, and assessment.  International journal of medical informatics, 79(12):849–859.
  * [Beckwith et al., 2006]  Bruce A Beckwith, Rajeshwarri Mahaadevan, Ulysses J Balis, and Frank Kuo.  2006\.  Development and evaluation of an open source software tool for deidentification of pathology reports.  BMC medical informatics and decision making, 6(1):1.
  * [Berman, 2003]  Jules J Berman.  2003\.  Concept-match medical data scrubbing: how pathology text can be used in research.  Archives of pathology & laboratory medicine, 127(6):680–686.
  * [Dernoncourt et al., 2016]  Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits.  2016\.  De-identification of patient notes with recurrent neural networks.  arXiv preprint arXiv:1606.03475.
  * [DesRoches et al., 2013]  Catherine M DesRoches, Chantal Worzala, and Scott Bates.  2013\.  Some hospitals are falling behind in meeting ‘meaningful use’ criteria and could be vulnerable to penalties in 2015.  Health Affairs, 32(8):1355–1360.
  * [Douglas et al., 2004]  Margaret Douglas, Gari Clifford, Andrew Reisner, George Moody, and Roger Mark.  2004\.  Computer-assisted de-identification of free text in the mimic ii database.  In Computers in Cardiology, 2004, pages 341–344. IEEE.
  * [Douglass et al., 2005]  Margaret Douglass, Gari Cliffford, Andrew Reisner, William Long, George Moody, and Roger Mark.  2005\.  De-identification algorithm for free-text nursing notes.  In Computers in Cardiology, 2005, pages 331–334. IEEE.
  * [Fielstein et al., 2004]  Elliot M. Fielstein, Steven H. Brown, and Theodore Speroff.  2004\.  Algorithmic de-identification of VA medical exam text for HIPAA privacy compliance: Preliminary findings.  Medinfo, 1590.
  * [Filannino and Nenadic, 2015]  Michele Filannino and Goran Nenadic.  2015\.  Temporal expression extraction with extensive feature type selection and a posteriori label adjustment.  Data & Knowledge Engineering, 100:19–33.
  * [Friedlin and McDonald, 2008]  Jeff Friedlin and Clement J McDonald.  2008\.  A software tool for removing patient identifying information from clinical documents.  Journal of the American Medical Informatics Association, 15(5):601–610.
  * [Goldberger et al., 2000]  Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley.  2000\.  Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals.  Circulation, 101(23):e215–e220.
  * [Guo et al., 2006]  Yikun Guo, Robert Gaizauskas, Ian Roberts, George Demetriou, and Mark Hepple.  2006\.  Identifying personal health information using support vector machines.  In i2b2 workshop on challenges in natural language processing for clinical data, pages 10–11.
  * [Gupta et al., 2004]  Dilip Gupta, Melissa Saul, and John Gilbertson.  2004\.  Evaluation of a deidentification (De-Id) software engine to share pathology reports and clinical documents for research.  American journal of clinical pathology, 121(2):176–186.
  * [Hara, 2006]  Kazuo Hara.  2006\.  Applying a SVM based chunker and a text classifier to the deid challenge.  In i2b2 Workshop on challenges in natural language processing for clinical data, pages 10–11. Am Med Inform Assoc.
  * [Hochreiter and Schmidhuber, 1997]  Sepp Hochreiter and Jürgen Schmidhuber.  1997\.  Long short-term memory.  Neural computation, 9(8):1735–1780.
  * [Hsiao et al., 2011]  Chun-Ju Hsiao, Esther Hing, Thomas C Socey, and Bill Cai.  2011\.  Electronic health record systems and intent to apply for meaningful use incentives among office-based physician practices: United states, 2001–2011.  system, 18(17.3):17–3.
  * [Johnson et al., 2016]  Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  2016\.  Mimic-iii, a freely accessible critical care database.  Scientific data, 3.
  * [McCann, 2015]  Erin McCann.  2015\.  EHR vendor marketshare and MU attestations by vendor.  Healthcare IT News.
  * [Meystre et al., 2010]  Stephane M Meystre, F Jeffrey Friedlin, Brett R South, Shuying Shen, and Matthew H Samore.  2010\.  Automatic de-identification of textual documents in the electronic health record: a review of recent research.  BMC medical research methodology, 10(1):1.
  * [Morrison et al., 2009]  Frances P Morrison, Li Li, Albert M Lai, and George Hripcsak.  2009\.  Repurposing the clinical record: can an existing natural language processing system de-identify clinical notes?  Journal of the American Medical Informatics Association, 16(1):37–39.
  * [Neamatullah et al., 2008]  Ishna Neamatullah, Margaret Douglass, H Lehman Li-wei, Andrew Reisner, Mauricio Villarroel, William J Long, Peter Szolovits, George B Moody, Roger G Mark, and Gari D Clifford.  2008\.  Automated de-identification of free-text medical records.  BMC medical informatics and decision making, 8(1):1.
  * [Office for Civil Rights, 2002]  HHS Office for Civil Rights.  2002\.  Standards for privacy of individually identifiable health information. final rule.  Federal Register, 67(157):53181.
  * [Parker et al., 2011]  Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda.  2011\.  English gigaword fifth edition, linguistic data consortium.  Technical report, Technical Report. Linguistic Data Consortium, Philadelphia.
  * [Pennington et al., 2014]  Jeffrey Pennington, Richard Socher, and Christopher D Manning.  2014\.  GloVe: global vectors for word representation.  Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12:1532–1543.
  * [Ruch et al., 2000]  Patrick Ruch, Robert H Baud, Anne-Marie Rassinoux, Pierrette Bouillon, and Gilbert Robert.  2000\.  Medical document anonymization with a semantic lexicon.  In Proceedings of the AMIA Symposium, page 729. American Medical Informatics Association.
  * [Saeed et al., 2011]  Mohammed Saeed, Mauricio Villarroel, Andrew T Reisner, Gari Clifford, Li-Wei Lehman, George Moody, Thomas Heldt, Tin H Kyaw, Benjamin Moody, and Roger G Mark.  2011\.  Multiparameter intelligent monitoring in intensive care II (MIMIC-II): a public-access intensive care unit database.  Critical care medicine, 39(5):952.
  * [South et al., 2014]  Brett R South, Danielle Mowery, Ying Suo, Jianwei Leng, Óscar Ferrández, Stephane M Meystre, and Wendy W Chapman.  2014\.  Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text.  Journal of biomedical informatics, 50:162–172.
  * [Stubbs et al., 2015]  Amber Stubbs, Christopher Kotfila, and Özlem Uzuner.  2015\.  Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task track 1.  Journal of biomedical informatics, 58:S11–S19.
  * [Sweeney, 1996]  Latanya Sweeney.  1996\.  Replacing personally-identifying information in medical records, the Scrub system.  In Proceedings of the AMIA annual fall symposium, page 333. American Medical Informatics Association.
  * [Szarvas et al., 2006]  György Szarvas, Richárd Farkas, and András Kocsor.  2006\.  A multilingual named entity recognition system using boosting and c4.5 decision tree learning algorithms.  In Discovery Science, pages 267–278. Springer.
  * [Thomas et al., 2002]  Sean M Thomas, Burke Mamlin, Gunther Schadow, and Clement McDonald.  2002\.  A successful technique for removing names in pathology reports using an augmented search and replace method.  In Proceedings of the AMIA Symposium, page 777. American Medical Informatics Association.
  * [Uzuner et al., 2008]  Özlem Uzuner, Tawanda C Sibanda, Yuan Luo, and Peter Szolovits.  2008\.  A de-identifier for medical discharge summaries.  Artificial intelligence in medicine, 42(1):13–35.
  * [Wright et al., 2013]  Adam Wright, Stanislav Henkin, Joshua Feblowitz, Allison B McCoy, David W Bates, and Dean F Sittig.  2013\.  Early results of the meaningful use program for electronic health records.  New England Journal of Medicine, 368(8):779–780.
  * [Yang and Garibaldi, 2015]  Hui Yang and Jonathan M Garibaldi.  2015\.  Automatic detection of protected health information from clinic narratives.  Journal of biomedical informatics, 58:S30–S38.