MIDLMedical Imaging with Deep Learning 2023 Short Paper – MIDL 2023 submission
– Under Review Michelle IskandarContributed equally2 Harvey
ManneringContributed equally1 Zhanxiang SunContributed equally1 Jacqueline
Matthew 2 Hamideh Kerdegari 2 Laura Peralta2 Miguel XochicaleContributed
equally1
1 University College London 2 King’s College London

#  Towards Realistic Ultrasound Fetal Brain Imaging Synthesis

###### Abstract

Prenatal ultrasound imaging is the first-choice modality to assess fetal
health. Medical image datasets for AI and ML methods must be diverse (i.e.
diagnoses, diseases, pathologies, scanners, demographics, etc), however there
are few public ultrasound fetal imaging datasets due to insufficient amounts
of clinical data, patient privacy, rare occurrence of abnormalities in general
practice, and limited experts for data collection and validation. To address
such data scarcity, we proposed generative adversarial networks (GAN)-based
models, diffusion-super-resolution-GAN and transformer-based-GAN, to
synthesise images of fetal ultrasound brain planes from one public dataset. We
reported that GAN-based methods can generate 256x256 pixel size of fetal
ultrasound trans-cerebellum brain image plane with stable training losses,
resulting in lower FID values for diffusion-super-resolution-GAN (average 7.04
and lower FID 5.09 at epoch 10) than the FID values of transformer-based-GAN
(average 36.02 and lower 28.93 at epoch 60). The results of this work
illustrate the potential of GAN-based methods to synthesise realistic high-
resolution ultrasound images, leading to future work with other fetal brain
planes, anatomies, devices and the need of a pool of experts to evaluate
synthesised images. Code, data and other resources to reproduce this work are
available at https://github.com/budai4medtech/midl2023.

###### keywords:

Medical Image Synthesis, Ultrasound Fetal Imaging, GANs

††editors: Under Review for MIDL 2023

##  1 Introduction

Prenatal imaging is performed to assess various aspects of pregnancy,
including confirmation of the pregnancy, screening for developmental defects,
and investigation of pregnancy complications (Kline–Fath and Bitters, 2007).
In the last decade, the fields of machine learning (ML) and artificial
intelligence (AI) have been successful to model intelligent behaviors with
minimal human interference (Hamet and Tremblay, 2017). Particularly, automatic
classification of fetal ultrasound planes and fetal head biometric measurement
(Burgos-Artizzu et al., 2020b; Sin, 2018; Fiorentino et al., 2022). Despite
such advances, there are few challenges faced in prenatal imaging: (a) the
accuracy of recorded measurements which can be caused by differences in intra-
view variability of imaging equipment and inter-observer variability of
sonographer skills (England, 2015; Sarris et al., 2012; Villar et al., 1989;
Kesmodel, 2018), (b) availability of expert clinicians or trained technicians
to select, to classify and to validate regions of interest (Burgos-Artizzu et
al., 2020a), (c) the insufficient and limited amount of clinical data (Jang et
al., 2018; Sin, 2018; He et al., 2021), (d) data accessibility due to patient
privacy or protection of personal health information (Shin et al., 2018), and
(e) the cost of acquisition of clinical data as it requires expensive imaging
equipment and experts for data collection and validation (Wang et al., 2019;
Kim et al., 2019). Given the advances with generative adversarial networks
(GAN) methods to handle problems in medical reconstructions, image resolution,
enhancement, segmentation, lesion detection, data simulation or classification
(AlAmir and AlGhamdi, 2022), we hypnotise that realistic ultrasound imaging
can address challenges in data scarcity, accessibility and expensiveness. For
instance, Eli et al. (2017) proposed a method of generating freehand
ultrasound image simulation using a spatially conditioned GAN. Kazeminia et
al. (2020) presented a review of the state-of-the-art research in GAN in
medical imaging for classification, denoising, reconstruction, synthesis,
registration, and detection. Montero et al. (2021) proposed a method to
generate fetal brain US images using an unconditional GAN, StyleGAN2,
specifically to improve the fine-grained plane classification, specifically
the trans-thalamic and trans-ventricular plane. Hence, the aim of this work is
to show the potential of GAN-based methods to generate realistic ultrasound
fetal trans-cerebellum brain plane imaging with small datasets.

##  2 Methods and datasets

###  2.1 Diffusion-Super-Resolution-GAN (DSR-GAN)

We use a Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020)
followed by a Super-Resolution-GAN (Ledig et al., 2017). To reduce computation
time, we finetune a pretrained DDPM to produce 128x128 pixel images and then
scale them up to 256x256 using SRGAN. After the DDPM and before SRGAN,
histogram matching (Castleman, 1996) is applied to ensure that the colour
distribution of the synthetic images matches the colour distribution of the
real images.

###  2.2 Transformer-based-GAN (TB-GAN)

We use StyleSwin model, which features Swin Transformer layers designed to
capture high quality details of the original images while simultaneously
reducing memory usage, enabling synthesized images of higher resolutions Zhang
et al. (2022). Differentiable data augmentation (DiffAug) and adaptive pseudo
augmentation (APA) are implemented to combat discriminator over-fitting due to
limited data and ensure stability in training process Zhao et al. (2020);
Jiang et al. (2021).

###  2.3 Image Quality Assessment

Quality of synthesised images are evaluated with Fréchet inception distance
(FID), measuring the distance between distributions of synthesised and
original images (Heusel et al., 2017). The lower the FID number is, the more
similar the synthesised images are to the original ones. FID metric showed to
work well for fetal head ultrasound images compared to other metrics Bautista
et al. (2022).

###  2.4 Datasets

Trans-cerebellum brain plane ultrasound images from Voluson E6 were used for
this work, consisting of 408 training images (Burgos-Artizzu et al., 2020a,
b). Scans were collected by multiple operators of similar skill level at
BCNatal during standard clinical practice between October 2018 and April 2019.
DICOM images were collected and anominsied using png format, resulting in
images of various pixel size (e.g., 692x480, 745x559, and 961x663).

##  3 Experiments: Design and results

Diffusion model was finetuned for 10000 epochs with the Adam optimiser to then
train SRGAN for 200 epochs from scratch with the Adam optimiser. The images
used to train both models are flipped horizontally, zoomed and rotated
randomly to increase the variety of the dataset(Fig 1c). Transfer learning is
used when training StyleSwin. The model was firstly pre-trained on Trans-
thalamus plane for 500 epochs as contains larger number of images (1072).
Then, the model was fine-tuned on Trans-cerebellum plane images for an
additional 200 epochs. Adam optimizer was also used during both pre-training
and fine-tuning stages, following the two time-scale update rule with learning
rates of 1e-4 for the discriminator and 1e-5 for the generator van den Heuvel
et al. (2018).

Figure 1:  Results from Diffusion-Super-resolution-GAN (DSR-GAN) and
transformer-based-GAN (TB-GAN): (a) Training losses for Generator and
Discriminator networks, (b) FID scores, and (c) 256x256 pixel size trans-
cerebellum images of two randomised batches (B1, B2) of real and synthesised
(DSR-GAN and TB-GAN).

##  4 Conclusions and future work

Synthesising fetal brain images with the diffusion-Super-Resolution-GAN and
transformer-based-GAN methods were successful, generating images of 256x256
pixel size resolution with stable loss values and resulting in lower FID
values for diffusion-super-resolution-GAN (average 7.04 and lower FID 5.09 at
epoch 10) compared to FID values of transformer-based-GAN (average 36.02 and
lower 28.93 at epoch 60). Such results suggest, as future work, the potential
to synthesise realistic higher-resolution fetal ultrasound images for other
anatomies and ultrasound-devices.

## References

  * Sin (2018) _Human-level Performance On Automatic Head Biometrics In Fetal Ultrasound Using Fully Convolutional Neural Networks_ , 7 2018. IEEE.  ISBN 978-1-5386-3646-6.  10.1109/EMBC.2018.8512278.
  * AlAmir and AlGhamdi (2022) Manal AlAmir and Manal AlGhamdi.  The role of generative adversarial network in medical image analysis: An in-depth survey.  _ACM Comput. Surv._ , mar 2022.  ISSN 0360-0300.  10.1145/3527849.  URL https://doi.org/10.1145/3527849.  Just Accepted.
  * Bautista et al. (2022) Thea Bautista, Jacqueline Matthew, Hamideh Kerdegari, Laura Peralta Pereira, and Miguel Xochicale.  Empirical study of quality image assessment for synthesis of fetal head ultrasound imaging with dcgans, 2022.  URL https://arxiv.org/abs/2206.01731.
  * Burgos-Artizzu et al. (2020a) Xavier P. Burgos-Artizzu, David Coronado-Gutiérrez, Brenda Valenzuela-Alcaraz, Elisenda Bonet-Carne, Elisenda Eixarch, Fatima Crispi, and Eduard Gratacós.  Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes.  _Scientific Reports_ , 10(1):10200, Jun 2020a.  ISSN 2045-2322.  10.1038/s41598-020-67076-5.  URL https://doi.org/10.1038/s41598-020-67076-5.
  * Burgos-Artizzu et al. (2020b) Xavier P. Burgos-Artizzu, David Coronado-Gutierrez, Brenda Valenzuela-Alcaraz, Elisenda Bonet-Carne, Elisenda Eixarch, Fatima Crispi, and Eduard Gratacós.  Fetal planes db: Common maternal-fetal ultrasound images.  _Scientific Reports_ , 6 2020b.  10.5281/ZENODO.3904280.  URL https://zenodo.org/record/3904280.
  * Castleman (1996) Kenneth R Castleman.  _Digital image processing_.  Prentice Hall Press, 1996.
  * Eli et al. (2017) Eli, Lee Li-Lin, Xie Weidi, Barratt Dean C., Vercauteren Tom, Noble J Alison Hu Yipeng, and Gibson.  Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks.  _Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment_ , pages 105–115, 2017.
  * England (2015) NHS England.  _Fetal anomaly screening programme handbook_.  NHS Digital, 7 2015.
  * Fiorentino et al. (2022) Maria Chiara Fiorentino, Francesca Pia Villani, Mariachiara Di Cosmo, Emanuele Frontoni, and Sara Moccia.  A review on deep-learning algorithms for fetal ultrasound-image analysis, 2022.  URL https://arxiv.org/abs/2201.12260.
  * Hamet and Tremblay (2017) Pavel Hamet and Johanne Tremblay.  Artificial intelligence in medicine.  _Metabolism_ , 69:S36–S40, 4 2017.  ISSN 00260495.  10.1016/j.metabol.2017.01.011.
  * He et al. (2021) Fujiao He, Yaqin Wang, Yun Xiu, Yixin Zhang, and Lizhu Chen.  Artificial intelligence in prenatal ultrasound diagnosis.  _Frontiers in Medicine_ , 8, 2021.  ISSN 2296-858X.  10.3389/fmed.2021.729978.  URL https://www.frontiersin.org/article/10.3389/fmed.2021.729978.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in Neural Information Processing Systems_ , 30, 2017.  URL https://proceedings.neurips.cc/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 6840–6851. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf.
  * Jang et al. (2018) Jaeseong Jang, Yejin Park, Bukweon Kim, Sung Min Lee, Ja-Young Kwon, and Jin Keun Seo.  Automatic estimation of fetal abdominal circumference from ultrasound images.  _IEEE Journal of Biomedical and Health Informatics_ , 22:1512–1520, 9 2018.  ISSN 2168-2194.  10.1109/JBHI.2017.2776116.
  * Jiang et al. (2021) Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy.  Deceive d: Adaptive pseudo augmentation for gan training with limited data, 2021.
  * Kazeminia et al. (2020) Salome Kazeminia, Christoph Baur, Arjan Kuijper, Bram van Ginneken, Nassir Navab, Shadi Albarqouni, and Anirban Mukhopadhyay.  Gans for medical image analysis.  _Artificial Intelligence in Medicine_ , 109:101938, 2020\.  ISSN 0933-3657.  https://doi.org/10.1016/j.artmed.2020.101938.  URL https://www.sciencedirect.com/science/article/pii/S0933365719311510.
  * Kesmodel (2018) Ulrik S. Kesmodel.  Information bias in epidemiological studies with a special focus on obstetrics and gynecology.  _Acta Obstetricia et Gynecologica Scandinavica_ , 97:417–423, 4 2018.  ISSN 00016349.  10.1111/aogs.13330.
  * Kim et al. (2019) Mingyu Kim, Jihye Yun, Yongwon Cho, Keewon Shin, Ryoungwoo Jang, Hyun jin Bae, and Namkug Kim.  Deep learning in medical imaging.  _Neurospine_ , 16:657–668, 12 2019.  ISSN 2586-6583.  10.14245/ns.1938396.198.
  * Kline–Fath and Bitters (2007) Beth Kline–Fath and Constance Bitters.  Prenatal imaging.  _Newborn and Infant Nursing Reviews_ , 7:197–204, 12 2007\.  ISSN 15273369.  10.1053/j.nainr.2007.09.002.
  * Ledig et al. (2017) Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Montero et al. (2021) Alberto Montero, Elisenda Bonet-Carne, and Xavier Paolo Burgos-Artizzu.  Generative adversarial networks to improve fetal brain fine-grained plane classification.  _Sensors (Basel, Switzerland)_ , 21, 11 2021.  ISSN 1424-8220.  10.3390/s21237975.
  * Sarris et al. (2012) I. Sarris, C. Ioannou, P. Chamberlain, E. Ohuma, F. Roseman, L. Hoch, D. G. Altman, and A. T. Papageorghiou.  Intra- and interobserver variability in fetal ultrasound measurements.  _Ultrasound in Obstetrics and Gynecology_ , 39:266–273, 3 2012.  ISSN 09607692.  10.1002/uog.10082.
  * Shin et al. (2018) Hoo-Chang Shin, Neil A Tenenholtz, Jameson K Rogers, Christopher G Schwarz, Matthew L Senjem, Jeffrey L Gunter, Katherine P Andriole, and Mark H Michalski.  Medical image synthesis for data augmentation and anonymization using generative adversarial networks.  _ArXiv_ , abs/1807.10225, 2018.
  * van den Heuvel et al. (2018) Thomas L. A. van den Heuvel, Dagmar de Bruijn, Chris L. de Korte, and Bram van Ginneken.  Automated measurement of fetal head circumference using 2D ultrasound images.  _PLOS ONE_ , July 2018.  10.5281/zenodo.1327317.  URL https://doi.org/10.5281/zenodo.1327317.
  * Villar et al. (1989) J Villar, J Repke, L Markush, W Calvert, and G Rhoads.  The measuring of blood pressure during pregnancy.  _American journal of obstetrics and gynecology_ , 161:1019–24, 10 1989.  ISSN 0002-9378.  10.1016/0002-9378(89)90777-1.
  * Wang et al. (2019) Ruoyao Wang, Zhenghan Fang, Jiaqi Gu, Yi Guo, Shicong Zhou, Yuanyuan Wang, Cai Chang, and Jinhua Yu.  High-resolution image reconstruction for portable ultrasound imaging devices.  _EURASIP Journal on Advances in Signal Processing_ , 2019:56, 12 2019.  ISSN 1687-6180.  10.1186/s13634-019-0649-x.
  * Zhang et al. (2022) Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo.  Styleswin: Transformer-based gan for high-resolution image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11304–11314, June 2022.
  * Zhao et al. (2020) Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training, 2020.


J

<EMAIL_ADDRESS>Ratcliff II Kienzle

[a]NIST Center for Neutron Research 100 Bureau Drive, Gaithersburg MD USA
20899

#  _reductus_ : a stateless Python data-reduction service with a browser
frontend

William  Paul

###### Abstract

The online data reduction service _reductus_ transforms measurements in
experimental science from laboratory coordinates into physically meaningful
quantities with accurate estimation of uncertainties based on instrumental
settings and properties. This reduction process is based on a few well-known
transformations, but flexibility in the application of the transforms and
algorithms supports flexibility in experiment design, supporting a broader
range of measurements than a rigid reduction scheme for data. The user
interface allows easy construction of arbitrary pipelines from well-known data
transforms using a visual dataflow diagram. Source data is drawn from a
networked, open data repository. The Python backend uses intelligent caching
to store intermediate results of calculations for a highly responsive user
experience. The reference implementation allows immediate reduction of
measurements as they are recorded for the three neutron reflectometry
instruments at the NIST Center for Neutron Research (NCNR), without the need
for visiting scientists to install additional software on their own computers.

A web-based, flexible scientific data reduction system is presented, which
accesses published data stores and transforms raw measurements into
interpretable data through data-flow diagrams which are converted to advanced
calculations on a Python backend; the results are returned in real time
through the web interface. The application was developed for handling neutron
and X-ray reflectometry results at a user facility.

##  1 Motivation

The transformation of raw measurement output into meaningful, interpretable
data with attached uncertainties (data reduction) is a ubiquitous task in the
experimental sciences. In the case where the workflow is well-established and
the community is small the most direct way to accomplish this is to develop a
custom application to be installed on a limited number of dedicated computers.
However, at a scientific user facility a large number of visiting researchers
are using the measurement tools on a part-time basis. As such, it is necessary
to make reduction capabilities widely available and flexible. In this case a
web-based application is an attractive alternative to distributing dedicated
installable executables.

The main benefit of a web application is the almost universal accessibility to
potential users. On top of this, a centralized reduction server also benefits
from the ability to update the calculation code at any time without requiring
all users to update their software, as well as largely eliminating the non-
trivial time cost of maintaining an installable application for more than one
target platform. Users can also install the service locally much as they would
if reduction were a desktop application. This allows them to access their own
data and develop reduction applications for their own instrumentation.

The specific implementation described herein was developed to provide
reduction for reflectometry instruments, but the system was designed to be
extensible and usable for other data processing problems and work is underway
to support reduction for off-specular reflectometry, small-angle neutron
scattering (SANS), and triple axis spectrometry (TAS). The approach can be
adapted to any type of computational problem, but the level of flexibility and
computational complexity supported is driven by the needs of a data reduction
engine.

###  1.1 Data reduction for reflectometry

The measurements for reflectometry are performed at a particular configuration
in laboratory coordinates (primarily incident and detector angles) as recorded
in the raw data files, with some uncertainty in the precise values.[dura2006,
salah2007] In counting experiments with a beam of particles incident on the
sample there are many independent events for each configuration, each with a
slightly different wavelength and flight path. When accumulated, this
variation manifests as instrument resolution, which can be convolved with the
theoretical model of the experiment to produce an expected count rate for that
configuration. Usually the experiment will be set up so that configuration
uncertainty is much less than instrument resolution in order to maximize the
information gained from the measurement, and so configuration uncertainty is
largely ignored.

In order to determine the relative portion of scattered counts for each
instrument configuration, the total incident beam rate must be measured
independently, then simple division can normalize to a relative count rate.
With the incident beam generated according to a Poisson distribution, the
uncertainty in counts grows as $\surd n$, which leads to count rate estimates
which grow as $1/\surd n$; counting longer leads to smaller error bars.

There will be some background count rate, either from alternative paths to the
detector such as scattering through air, or from completely distinct processes
such as electronic noise, which can be subtracted from the measured scattering
signal giving an estimate of the true signal. Background subtraction can lead
to negative values, which is an unphysical estimate of the true count rate. A
more complete analysis of the data would model both the signal and background,
and determine parameter estimates consistent with both the background
measurement alone and with the combined signal and background measurement. By
exposing the complete reduction chain the user is free to choose the style of
analysis.

##  2 Dataflow diagram as a template for computation

The reduction process can be described as a series of data transformations,
with data flowing from one transformation module into the next. Corrections
such as background subtraction and incident rate normalization can be chained
together in a dataflow diagram,[sutherland1966] with the final data set saved
as the result of the reduction. This yields the estimated count per incident
count for each configuration. The incident angles and wavelengths converted to
more meaningful reciprocal space $q$ (inverse distance) coordinates (see
Figure 1). The individual modules may have control parameters, such as a scale
factor for a scaling module (needed if there is an attenuator in one
measurement but not in the others).

Figure 1:  (a) Measured signal and background counts. (b) Dataflow diagram,
with the left boxes for each node representing the input data and the right
boxes representing the possible outputs. The experimentalist can select the
signal, background and normalization as input files, producing the reduced
data as the output for the divide node. (c) Reduced output.

###  2.1 Data types

The data flowing between modules has a type associated with it. In order to
connect the output of one module to the input of the next module, the type of
the output parameter on the first module must match the type of the input
parameter on the subsequent module. For each data type, there are methods to
store and load the data, to convert the data to display form, and to export
the data. The stored format should be complete, so that reloading saved data
returns an equivalent data set. The displayed and exported forms might be a
subset of the total data.

###  2.2 Operations

By implementing the transformation modules in Python, the instrument scientist
has access to a large library of numerical processing facilities based on
NumPy and SciPy,[oliphant2007python] and in particular, libraries which
support propagation of uncertainties.[lebigot2013uncertainties] The _reductus_
library includes additional facilities for unit conversion, data rebinning,
interpolation and weighted least squares solving as well as its own simplified
uncertainties package for handling large data sets.

###  2.3 Bundles of inputs

It is often the case that many measurements need to be combined, with the same
computation steps applied to each data file. Rather than defining a separate
chain for each file, _reductus_ instead sends bundles of files between nodes.
To interpret the bundles, the module parameters are defined as either single
or multiple. If the primary input is single, then the module action operates
on each of the inputs separately; if multiple, then all inputs are passed to
the module action as a single list. For example, if several measurements must
be scaled independently then joined together into a single data set, both the
scale module input would be tagged as single, but the join module input would
be tagged multiple. The scale factor would be tagged multiple, indicating a
separate scale factor for each input. Outputs can also be single or multiple.
Unlike the join module, which produces a single output from multiple inputs, a
split module would produce multiple outputs from a single input. A module
which rebalances data amongst inputs (e.g., to correct for leakage between
polarization states in a polarized beam experiment), takes multiple inputs and
produces multiple outputs.

###  2.4 Instrument and module definition

An instrument is a set of data types and the computation modules for working
with them. A computation module has a number of properties, including name,
description, version, module action and parameters. Each parameter has an id,
a label, a type, a description, and some flags indicating whether the
parameter is optional or required, and if it is single or multiple. Input and
output parameters use one of the data types defined for the instrument.
Control parameters can have a variety of types, including simple integers,
floats or strings, or more complicated types such as indices into the data set
or coordinates on the graph, allowing some parameter values to be set with
mouse pointer interaction in the user interface.

###  2.5 Module interface language

To encourage accurate and complete module documentation, the module interface
can be extracted from the stylized module documentation and function
definition. The module interface language starts with an overview of the
module action. For each input, control and output parameter it gives the data
type and units and provides a short description of the parameter which can be
displayed as a tool tip in the user interface. Inputs and control parameters
are distinguished by examining the action declaration; the positional
parameters are inputs that can be wired to another node’s output and the
keyword parameters are control parameters. After the parameters, the module
documentation should define the author and version. The module name is set to
the name of the action function.

The module interface language is valid ReStructured Text (RST), which means
that the standard docutils toolset for Python can be used to convert the
documentation string to hypertext markup (HTML) or portable document format
(PDF). The conversion to HTML is performed with Sphinx, allowing for the
creation of an independent user manual for each instrument; it is also done
dynamically for each module for display in the user interface. Embedded
equations are rendered in HTML using mathjax, a TeX equation interpreter for
javascript.

###  2.6 Serialization of the diagram

A dataflow diagram is represented as a list of nodes numbered from $0$ to $n$,
with each node having a computation module, a label, an $(x,y)$ position, and
values for the control parameters of the computation module. The connections
are defined as a list of links, with each link having a source (node number
and output parameter name), and a target (node number and input parameter
name).

Every diagram can be used as a template, with the configuration values for the
nodes packaged separately from the diagram. The computation engine looks first
in the configuration for control parameter values for the node, using the
value given in the diagram if a specific configuration value is not found. If
no value is provided in the configuration or in the diagram then the default
parameter value for the module is used.

##  3 Backend

The backend is composed of two main pieces: a traditional web (HTTP) server
providing static resources including HTML, javascript, and cascading style
sheet (CSS) source code for the client application in the user’s browser, and
a computation engine that handles requests for reduction calculations as
remote procedure calls (RPC), as diagrammed in Fig. 2. A pool of shared
calculation engines is shown but is only recommended for a production server,
as a single engine is sufficient for a single-user test environment. A shared,
disk-backed computation cache is not required but is strongly recommended for
a responsive server, even in a single-user environment (a per-instance, non-
persistent in-memory cache is the fallback option.)

The Data Store in Fig. 2 is not part of the server but is an HTTP-accessible
source of raw data111for example, the NIST Center for Neutron Research data
store is located at https://dx.doi.org/10.18434/T4201B which is loaded as the
first step of a reduction. This arrangement makes it possible to do data
reduction without handling the raw data files on the client computer—the user
can download just the reduced data if they wish.

Figure 2: _reductus_ system diagram. Upon receiving a request from the User
Interface, the Load Balancer on the Web server will find an available Python
thread to run the reduction diagram. The first step will be to fetch the
requested data files from the data source and save them in the Redis Cache.
Intermediate calculations may also be cached allowing future repeated requests
to be returned immediately to the client, trading efficiency against the size
of the cache. As demand increases the different parts can be run on different
servers to spread the load.

###  3.1 Computation server

The point of contact for the client is the web server, which serves the static
resources (HTML, JS, CSS) as well as being a proxy gateway to the calculation
engines through the Python web services gateway interface (WSGI).

####  3.1.1 Converting the diagram to computations

A dataflow diagram is a directed acyclic graph (DAG), with the modules as the
nodes in the graph and the connections from outputs to inputs as the links
between the nodes. No cycles are allowed, which means that the output of a
module cannot be used to compute its own input. Every DAG has topological
order, so the nodes can be arranged linearly such that the independent nodes
appear first, and each dependent node appears after all of its input nodes. By
computing nodes in topological order all inputs are guaranteed to be computed
before they are needed. Although there are linear time algorithms for sorting
a DAG, the diagram sizes for data reduction are small enough that a naïve
$O(n^{2})$ algorithm can be used.

####  3.1.2 Results and source caching

The results of every calculation request are cached on the server. There are
several choices in the configuration of the server but the default is to use a
Redis[sanfilippo2012redis] key-value store with a Least-Recently-Used (LRU)
expiry algorithm and a maximum cache size. This can be started automatically
as a system service at startup on a Linux server, allowing worry-free
operation between server reboots.

The server cache is very important to the performance of the service: the
slowest part of many computations is retrieving the source data files from the
data repository to the server. With caching, this typically will only happen
once per data file per experiment, and after that the server-local cached
version will be used. For data files, it is assumed that they can be uniquely
identified by their resource path (URL) and last-modified time, (accurate to a
second due to the file system mtime limitations). It is therefore fallible if
the file is rewritten in a time span of less than a second, but the data files
we are using are updated much more slowly than that.

Each calculation step is identified by a unique id, created by hashing all of
the input values along with the version number of the code for the step
itself. For inputs which are the result of a previous calculation step, the
hash of that step is used to identify the input. Since the calculations on the
same data should give the same results each time, the results can be cached
using this key and retrieved for future calculation steps or returned to the
web client for display. If the calculation parameters change (for example, the
scale factor for a scaling step), then the hash will change, and the step will
be recalculated and cached with the new key. This will change the input values
for the subsequent step, which will change its hash, which will cause it to be
recalculated as well. In this way, if the source data (timestamp) changes, all
reduction steps which depend upon it will be updated. By including the version
number of each step in the cache, changes to the server will automatically
cause all future reductions to be recomputed, even if they are already cached.

####  3.1.3 Data Provenance and Reproducibility

In a highly interactive environment, where parameters, files and even work
flows can be modified and the results saved at any time, it is important to
have a record of the inputs as well as the results.[simmhan2005] Therefore,
the reduction template and all its parameters are stored along with the
reduced data in each saved file. By loading a file into _reductus_ the precise
steps required to reproduce the data become visible. The NCNR data source is
referenced through a Digital Object Identifier (DOI), with the implicit
promise of a stable reference even if the data is moved to a new URL, thus
providing long term reproducibility of the reduction.

The server uses the current version of each reduction step to evaluate the
outputs. This effectively acts as a behind-the-scenes update to all steps in
the reduction process; any steps that are newer will be recomputed, and the
updated results can be re-exported. This is particularly useful for reductions
performed during data acquisition. As newer measurements are added the updated
timestamp will force recomputation of all subsequent steps.

In order to reproduce an existing reduction, the server version at the time of
the reduction must be used. The server source is managed with the git source
control system,[torvalds2010git] available on GitHub at
https://github.com/reductus/reductus. Git creates a unique hash of the entire
source tree for each commit which is stored as part of the template. To
reproduce the data from a specific reduction this hash must be used to
retrieve the source code and run it as a local server. The specific versions
of the dependencies (scipy, numpy, uncertainties, $\ldots$) can be recorded in
the source tree as well to protect against changing interfaces. Because users
can easily revert to older versions of the software, developers are free to
modify the code at will and maintain a clean code base.

####  3.1.4 Statelessness

The computation engine maintains no state. The user interface manages the
interactions of the end-user with the engine, and keeps a working copy of the
active dataflow template(s); the browser session is the only real context.
This has distinct operational advantages for the compute engine—it can be
restarted at any time with close to zero impact on the availability and
continuity of the service. The cache is persistent between engine restarts,
but can be completely wiped if necessary without destroying user data
(performance will suffer temporarily as the calculations are re-cached).

###  3.2 Server configurations

The system is designed to be modular, allowing a number of possible
configurations of the needed resources.

####  3.2.1 Simple single-computer configuration

The simplest configuration is to run the web server, calculation nodes and
cache on the same computer. A server implementation using the Python flask
package is provided, which can simultaneously serve the static client
resources as well as the RPC calculation requests. This server is suggested
for use when running a private reduction service.

A private server is required for processing data sets stored locally; since
the service is stateless, providing neither data upload nor storage of
reduction diagrams and reduced data, there is no other way to reduce data that
is not present in the public data repositories.

Local file access is only enabled for servers running on the private
“localhost” network interface. Such servers should not allow external
connections since access to local files is a security risk.

Similarly, a private server is required for custom reduction steps that are
not part of the standard service since the standard service does not allow the
execution of arbitrary Python code. Users at facilities that do not allow
external access to the web will need to copy all the data files to a local
machine and reduce it with a private server.

####  3.2.2 Container-based configuration

The service can be run using a linux container environment. A recipe for
Docker is provided in the source code for running the application as three
coordinated containers: one for the web server, one for the Python calculation
engine and one for the Redis cache. The current snapshot of the source code in
the users’ directory is copied into the containers as part of the build step,
so this is a useful setup for development and testing. The user need only
install Docker and Docker-Compose. The supporting tools, including Python,
Redis and all dependent libraries are pulled in by the Docker-Compose recipe.
This greatly eases ability of users to extend the project and to test new
features.

####  3.2.3 Scalable production server configuration

For production (public-facing) servers, the static files can be copied from
the /static subfolder of the source repository to a web server such as Apache
or nginx, where requests to the /RPC2 path are forwarded to a pool of Python
calculation engines (e.g., using uwsgi to run the calculations with Apache
mod_uwsgi_proxy acting as the load balancer), sharing a Redis instance for
caching.

An elastic on-demand service could be built from these components as well,
with multiple (replicated) Redis caches and no limit on the number of worker
Python calculation engines that can be employed by a High-Availabilty web
server setup. The statelessness of the server means that no complicated
synchronization is required between server components.

##  4 Web interface

The user interface for _reductus_ is a javascript application that runs in a
browser. The application relies on a number of advanced javascript libraries,
and as such it is supported only by browsers that have reasonably standards-
compliant implementations of javascript (ECMAScript version $\geq 5$). The
application is made up of these key components (see Fig. 3):

  * •

visual, interactive dataflow diagram

  * •

panel for setting parameters for individual computation modules

  * •

plotting panel to show results

  * •

data source file browser

Figure 3: _reductus_ user interface. Panels (clockwise, from upper left):
source data browser, interactive dataflow diagram, and current module
parameters; in the center is a plot of the data corresponding to the current
module.

The dataflow display and editor, plotting routines and parameter-setting
panels were implemented using the D3.js visualization library.[bostock2011-d3]

Once connected to the server, the web client can query the computation service
for the public data stores, the available reduction steps, and a set of
predefined template diagrams representing the usual reduction procedures for
the data. The menus and default options are populated based on this
information.

###  4.1 Dataflow diagram

The user interacts with the dataflow diagram in order to navigate the
reduction chain: by clicking on a module within the diagram to bring up the
parameters panel for that module, or by clicking on the input (left) or output
(right) terminals to display calculation results. Changes made in the
parameters panel are by default immediately applied to the active template,
though by un-checking the auto-accept parameters box in the data menu an
additional confirmation step (pressing the accept button in the panel) can be
imposed to avoid accidental changing of the active template. In that context,
if a user makes parameter change but then selects a different module without
first pressing accept, that change is lost.

The clear button removes all values from the active parameters; the client
then repopulates the panel with default values for each parameter.

The web client creates a javascript object notation (JSON) representation of
the dataflow diagram along with an indicator of the input or output of
interest, and sends the request to the server via HTTP POST. The response is a
plottable representation of the data for that connector encoded either as JSON
or MSGPACK, which is then displayed in the plotting panel.

Clicking on any output within the dataflow diagram (including the rightmost,
“final” output) will trigger a calculation of all ancestor results for that
result in the diagram on the server, as described in section 3.1.1.

So, while the user has the option to inspect intermediate calculation steps,
for a routine reduction dataflow one can simply populate the file inputs and
click on the final output for export the completely reduced data.

###  4.2 Parameters panel

At the beginning of a reduction, a user chooses an instrument definition for
working with the data. As described in 3.1.1, this includes a list of data
types and a list of reduction steps (modules) to act on those data types. The
parameters panel is rendered based on the definition for the chosen module
type, using the predefined types for each input field to the module function,
mapping the simple known types (int, float, bool, etc.) to HTML form elements.
Some field types have renderers with enhanced interaction with the plot panel,
such as an index type which allows clicking on data points to add them to an
index list in the parameters panel. Another example of enhanced interactions
is the scale type which enables dragging a whole dataset on the plot to set
the scaling factor in the parameters panel.

When parameters are changed in this panel and committed with accept, they will
be used in any calculation of data flowing through that module.

###  4.3 Browser caching of calculations

In addition to the caching provided on the server for avoiding recalculation
of identical results, a local browser cache of calculations is provided. This
is particularly useful for the initial source data load, in which metadata
from all of the files in a source directory are passed to the client for
inspection and sorting in the source file browser. Naturally in a data
reduction scheme, the amount of data on the input side (loaders) is much
greater than the output result, so caching of the inputs helps tremendously
when making small adjustments interactively to the dataflow algorithm or
parameters.

###  4.4 Sessions and persistence

The _reductus_ server is stateless; the reduction diagrams created by the user
are not stored (a unique hash of the template representation may be associated
with cached calculations on the server, but no user template is ever stored
there). The only state associated with a session is stored in the browser or
on the filesystem of the user’s computer.

####  4.4.1 Stashing in-browser

Results of calculations can be “stashed” in the local persistent memory of the
browser. A list of these results can be recalled in the client and reused in
two ways: by reloading the entire calculation into the active dataflow panel,
or by selecting multiple stashed results to directly compare their outputs.

####  4.4.2 Saving to and loading from filesystem

In addition to the local browser store, the user may download a text version
of the dataflow diagram in JSON format with
Menu$\rightarrow$Template$\rightarrow$Download, which can be reloaded with
Menu$\rightarrow$Template$\rightarrow$Upload. The file contains the diagram
along with any field values, including the names of the input files. The
actual data is not included.

The data for the currently selected node can be saved with
Menu$\rightarrow$Data$\rightarrow$Export. This prompts the user for a
filename, then produces a tab-delimited column-format text file with the
dataflow diagram prepended as a comment header. The stored diagram allows a
full reduction to be reloaded into the client with
Menu$\rightarrow$Data$\rightarrow$Reload Exported (note that this may trigger
recalculation if the raw data has been updated since the reduction was
exported).

Due to security limitations built into all current browsers the data may only
be saved to the user’s “Downloads” folder, while uploads can of course
originate from any user-readable folder.

####  4.4.3 Sharing data among collaborators

The dataflow diagram is self contained. The reduced-data text files produced
by the _reductus_ system can be shared with others easily by email or portable
media, and provide both useful data and a recipe in the header for recreating
the data from known sources with known transformations; the chain of logic can
be inspected and verified by reloading the dataflow into the web client at any
time, thus assuring data provenance. This allows for easy collaboration
amongst users without the need for accounts or passwords.

##  5 Conclusions

The _reductus_ system is an interesting experiment in providing stateful web
services with a stateless server. Although users lose the convenience of cloud
services for managing their data, they are free from the inconvenience of
maintaining yet another user ID and password. Files can be stored and shared
using familiar tools such as email. The server is easy to adapt and install
locally for the rare user that needs more than the rigid set of functions and
data sources provided in the remote web service; this is no more complex than
adapting and installing the equivalent desktop application would be. For the
developer, the stateless server needs very little maintenance. There are no
database migrations needed, no backups required, and moving the service to a
different computer is as simple as installing the software and redirecting the
domain name service (DNS) to an alternate IP. Javascript provides a flexible
environment for interactive applications, with a rich and growing ecosystem of
libraries that work across most browser platforms. The Python backend provides
an ecosystem for rapid development of numerical code. The web services
middleware gives us scalability with no additional effort.

Making the dataflow graph visible and modifiable increases flexibility without
increasing complexity. Users with simple reduction problems can enter their
data on the left of the graph and retrieve the results on the right, ignoring
all steps in between. If there are problems, they can examine inputs and
outputs at each step and identify the cause. Although 85% of reduction is
performed on-site during the experiment, there were over 150 external users
access the system from across the United States and Europe in 2017. Without
the need to install a local version of the software we have far fewer support
requests; now they only occur for unusual datasets.

Feedback from the users has been overwhelmingly positive, and the new system
has completely supplanted our old reduction software for the three neutron
reflectometry instruments at the NCNR, with work underway to adapt the system
to several new instrument classes at the facility.

Acknowledgments

Significant contributions to the early code were made by Brendan Rowan, Alex
Yee, Ophir Lifshitz and Elakian Kanakaraj as part of the NIST Summer High
school Internship Program (SHIP) and Joseph Redmon as part of the Summer
Undergraduate Research Fellowship (SURF), supported by NSF CHRNS grant
DMR-1508249. We are grateful to Prof. Robert Briber of the Dept. of Materials
Science and Engineering at the University of Maryland for providing
significant technical support and operating the servers for early deployments
of the _reductus_ service.

The identification of commercial products or services in this paper does not
imply recommendation or endorsement by the National Institute of Standards and
Technology, nor does it imply that the equipment or service used is
necessarily the best available for the purpose.

## References

  * [1] [Bostock _et al._]Bostock, Ogievetsky Heer2011bostock2011-d3 Bostock, M., Ogievetsky, V. Heer, J. 2011.  _IEEE Trans. Visualization & Comp. Graphics (Proc. InfoVis)_.
  * [2] [Dura _et al._]Dura, Pierce, Majkrzak, Maliszewskyj, McGillivray, Lösche, ODonovan, Mihailescu, Perez-Salas Worcester2006dura2006 Dura, J. A., Pierce, D. J., Majkrzak, C. F., Maliszewskyj, N. C., McGillivray, D. J., Lösche, M., ODonovan, K. V., Mihailescu, M., Perez-Salas, U. Worcester, D. L. 2006.  _Review of scientific instruments_ , 77(7), 074301–074301.
  * [3] Lebigot2013lebigot2013uncertainties Lebigot, E. 2013.  _URL http://pythonhosted.org/uncertainties_.
  * [4] Oliphant2007oliphant2007python Oliphant, T. E. 2007.  _Computing in Science & Engineering_, 9(3).
  * [5] [Salah _et al._]Salah, Harzallah van der Lee2007salah2007 Salah, F., Harzallah, B. van der Lee, A. 2007.  _Journal of Applied Crystallography_ , 40(5), 813–819.
  * [6] Sanfilippo Noordhuis2012sanfilippo2012redis Sanfilippo, S. Noordhuis, P. 2012.  _Redis: The Definitive Guide: Data modeling, caching, and messaging_.  O’Reilly & Associates.
  * [7] [Simmhan _et al._]Simmhan, Plale Gannon2005simmhan2005 Simmhan, Y. L., Plale, B. Gannon, D. 2005.  _SIGMOD Rec._ 34(3), 31–36.
  * [8] Sutherland1966sutherland1966 Sutherland, W. R. 1966.  _The on-line graphical specification of computer procedures._ Thesis, Massachusetts Institute of Technology.
  * [9] Torvalds Hamano2010torvalds2010git Torvalds, L. Hamano, J. 2010.  _URL http://git-scm.com_.
  * [10]


Dennis<EMAIL_ADDRESS>Loic<EMAIL_ADDRESS>Alexis<EMAIL_ADDRESS>Frederic
<EMAIL_ADDRESS>Normandie Univ, UNICAEN, ENSICAEN, CNRS
UMR GREYC
6 Boulevard Maréchal Juin, CS 45053, 14050 CAEN cedex 4, France Safran
Rue des jeunes Bois, 78117 Châteaufort Training from generated data

# Training face verification models from generated face identity data

###### Abstract

Machine learning tools are becoming increasingly powerful and widely used.
Unfortunately membership attacks, which seek to uncover information from data
sets used in machine learning, have the potential to limit data sharing. In
this paper we consider an approach to increase the privacy protection of data
sets, as applied to face recognition. Using an auxiliary face recognition
model, we build on the StyleGAN generative adversarial network and feed it
with latent codes combining two distinct sub-codes, one encoding visual
identity factors, and, the other, non-identity factors. By independently
varying these vectors during image generation, we create a synthetic data set
of fictitious face identities. We use this data set to train a face
recognition model. The model performance degrades in comparison to the state-
of-the-art of face verification. When tested with a simple membership attack
our model provides good privacy protection, however the model performance
degrades in comparison to the state-of-the-art of face verification. We find
that the addition of a small amount of private data greatly improves the
performance of our model, which highlights the limitations of using synthetic
data to train machine learning models.

##  1 Introduction

In recent times, the availability and power of large data sets has increased
rapidly in a variety of different communities. With this influx of data come
new challenges. Many data sets include sensitive information, in particular
medical, biometric and financial data. It is insufficient to de-identify these
data sets, as it has been shown that it is possible to re-identify entries in
the data set using comparison with third-party data sets (Henriksen-Bulmer and
Jeary, 2016). One solution to these privacy problems in the application of
training machine learning algorithms is to create a sanitised synthetic data
set in place of a real one. As there is no requirement that a data set
consists of real samples, the synthetic data set merely needs to have the same
statistical properties as its source.

There are then two matters of interest: firstly, how well does the synthetic
data set approximate the real data set that it replaces, and secondly, how
well does the sanitised data set protect the sensitive data of the real data
set. The first question can be evaluated by comparing the performance of an
algorithm trained on the synthetic data set with one trained on the underlying
real data set. We would like to have both methods to give similar levels of
performance, as this would show that for our intended application the two data
sets have similar statistical properties.

In terms of the second question, it is difficult to ascertain the level of
privacy gained by creating a synthetic data set. If a method is used which
employs $\epsilon-$differential privacy, then the privacy budget, $\epsilon$,
can be used as a metric for how well the data is protected, however it is not
always practical to use $\epsilon-$differential privacy methods when working
with image generation. Indeed, it was noted in Saxena and Cao (2021) that
generators trained in such a differentially-private way fail to maintain a
trade-off between sample quality and diversity. It has been shown in recent
work that under some assumptions, synthetic data samples generated using GANs
are inherently $(\epsilon,\delta)-$ differentially private (Lin et al., 2021).

For our work we consider the scenario where we would like a data set of faces
which is private in terms of face identity, with application to face
verification. The data sets commonly used for this task are composed of faces
from celebrity individuals, whose identities are supposed to be in public
domain. This is not always the case however, as the large number of identities
needed for training face recognition algorithms requires a loose definition of
celebrity, such that many of the members of the data sets object to their
inclusion (Prabhakar et al., 2003; Senior and Pankanti, 2011). Privacy
concerns become even more clear when we consider private databases such as
those owned by Google and Facebook (Naker and Greenbaum, 2017). To build our
synthetic data set we build on the StyleGAN2 architecture (Karras et al.,
2020). StyleGAN2 is a powerful Generative Adversarial Network (GAN) that when
trained on the FFHQ dataset (Karras, ) can produce photorealistic random faces
belonging to random identities. Using StyleGAN2, we propose a method which
allows us to generate multiple samples from a random identity and therefore
enables the creation of a face identity dataset. With part of the StyleGAN2
latent code fixed, we generate multiple images and penalise dissimilarity in
facial features using the pre-trained state-of-the-art face recognition
algorithm ArcFace (Deng et al., 2019). We evaluate our synthetic data set on
the task of training a face recognition algorithm.

Compared to models from real data sets, the model trained using our generated
synthetic data set shows a drop of performance in the task of face
recognition. We show however that better performance is achieved when our
synthetic data set is supplemented with an existing real data set. Using a
simple membership attack, we show that our model trained with synthetic data
provides some degree of privacy protection relative to the real data model.

##  2 Related work

The idea of using GANs to generate synthetic datasets is not new. It has been
explored, for example in Choi et al. (2017) where synthetic Electronic Health
Records are generated using a GAN. Their results show that the synthetic data
is comparable to the real data in terms of distribution statistics,
performance in predictive modeling tasks, and by verification by an expert
review. They observe an empirical privacy gain from their method. An
advancement of this idea was made by Jordon et al. (2018) with their algorithm
PATE-GAN, which quantified a privacy gain by introducing a layer of
differential privacy. Their algorithm was based around a "student" GAN
discriminator which was taught to label synthetic examples as "real" or "fake"
based on the predictions of an array of "parent" discriminators. The parent
discriminators are trained on sections of the real data. The student however
is trained on synthetic data which has been labelled real or fake using a
differentially private vote aggregation from the parent discriminators. The
generator is trained using only the differentially private discriminator, so
by the post-processing rule of differential privacy, the generated dataset has
the same privacy guarantees as the private discriminator (Dwork et al., 2014).
The results work well using a variety of different datasets. The method has
not be shown to work well with image datasets however, possibly due to the
higher complexity of the data.

Compared to PATE-GAN and other work which involves conditional GANs such as
Odena et al. (2017) and Trigueros et al. (2018), our work has the added
constraint that for our synthetic dataset we would like to create not just
synthetic samples but also synthetic classes (our identities). This issue has
been explored by Donahue et al. (2017) who propose Semantically Decomposed
GANs (SD-GANs) which encourage the disentanglement of the latent space of
GANs. Their GAN is trained using a latent code $z$ decomposed into $z_{1}$ and
$z_{2}$. Two images are then produced using a fixed $z_{1}$ and varying
$z_{2}$. A discriminator is then tasked with differentiating between the pair
of fake images and a pair of images from the real dataset belonging to a
single identity. Using a facial verification tool, the authors show that their
method is successful in producing images belonging to the same identity,
however in contrast with our work there is no investigation of its performance
in the task of training a face recognition algorithm.

##  3 Background

###  3.1 Generative Adversarial Networks

In a GAN a generative network $G$ takes an input $\mathbf{z}\in Z$ drawn from
a known distribution $P_{z}$ and produces $G(z)$, which for our case is an
image $I$. The generator learns to produce images which match those drawn from
a target distribution $P_{r}$ by way of a minmax game played between the $G$
and a discriminator $D$. In the most simple formulation this takes the form
of:

$\min_{G}\max_{D}V(G,D)=\mathbbm{E}_{\mathbf{x}\sim P_{r}}[\log
D(\mathbf{x})]+\mathbbm{E}_{\mathbf{z}\sim P_{z}}[\log(1-D(G(\mathbf{z}))].$
(1)

(Karras et al., 2019) introduced the idea of StyleGAN. In StyleGAN and its
successor StyleGAN2, the generator can be considered split into a style
network and a synthesis network. As with a traditional GAN, the style network
takes an input $\mathbf{z}$ drawn from a distribution $P_{z}$. The network
then maps $z$ to another higher dimensional space $W$, which are called
"styles". This higher dimensional latent code is then input to the synthesis
network to generate the image $I$. Due to the mixing of different styles
during training and path length regularisation, the $W$-space latent codes of
StyleGAN able to achieve better native disentanglement compared to a regular
GAN.

###  3.2 Face recognition networks

Face recognition is a task that is predominately performed using deep
convolutional neural networks to create an embedding from an input face. The
adjacent task of face verification (determining if two face images belong to
the same identity or not) can then be performed by assessing the distance
between two embeddings. There are two main methods for training this
embedding, either by directly learning an embedding, as in the case of the
triplet loss, or by training a multi-class classifier. The latter is the
method of choice in the algorithm ArcFace (Deng et al., 2019), which we use in
this paper.

##  4 Synthetic data set generation

The model we consider is similar to the work of Donahue et al. (2017) in their
SD-GANs, but applied to StyleGAN2. We begin with a generator $G$ that takes
two input latent codes, $z_{1}\in\mathbbm{R}^{512}$ and
$z_{2}\in\mathbbm{R}^{512}$, both drawn from a standard normal distribution.
The codes $z_{1}$ and $z_{2}$ are intended to correspond to identity and non-
identity features respectively. Both input codes are then passed through
separate fully connected layers which are trained independently. They are then
concatenated to form a layer of size $\mathbbm{R}^{1024}$ which is the reduced
to an embedding layer $\in\mathbbm{R}^{512}$ from another fully connected
layer. A schematic of the architecture is shown in Figure 1.

Figure 1: a. the architecture of StyleGAN2 and b. the new architecture that we
train to separate identity from other factors.

During training three samples are generated for each sample in the batch: an
anchor with completely random factors $z_{1,0}$ and $z_{2,0}$, another
realisation of the same identity $z_{1,0}$ with new other factors $z_{2,+}$
and a third image from a different identity $z_{1,-}$ with the same non-
identity factors $z_{2,0}$ as from the anchor.

The GAN is trained using almost the exact same method as in the original
StyleGAN2 with a size of 256x256 pixels and using the FFHQ dataset for real
images. The training considers only the anchor images to calculate GAN losses.

The loss however is augmented by a penalty which forces $z_{1}$ and $z_{2}$
into identity and non-identity factors using two auxiliary networks: a pose
network $P$ which, operating on a face, returns the 3-angle pose position of
the face (Ruiz et al., 2018), and the ArcFace face embedding network $F$ (Deng
et al., 2019) which returns a face identity embedding in $\mathbbm{R}^{512}$.
Designating the images as $I_{0}$ for anchor, $I_{+}$ for fixed identity and
$I_{-}$ for different identity images, the new losses which we introduce to
the existing GAN losses are:

$\begin{split}L_{P}^{-}=\lVert{P(I_{0})-P(I_{-})}\rVert^{2},&\>L_{P}^{+}=-\lVert{P(I_{0})-P(I_{+})}\rVert^{2}\\\
L_{F}^{-}=-\arccos{(F(I_{0})\cdot
F(I_{+}))},&\>L_{F}^{+}=\arccos{(F(I_{0})\cdot F(I_{-}))}.\end{split}$ (2)

These losses are added to generator loss (cf. Eq.(1)) with a weighting
strength of $\lambda=0.1$. The different losses are clamped so that they do
not effect intra-class diversity once $z_{1}$ and $z_{2}$ have been
sufficiently disentangled. The clamping is as follows:

$\begin{split}L_{P}^{+}=\max(L_{P}^{+},3.0),&\>L_{P}^{-}=\min(L_{P}^{-},5.0)\\\
L_{F}^{-}=\max(L_{F}^{-},0.70),&\>L_{F}^{+}=\min(L_{F}^{+},1.40).\end{split}$
(3)

##  5 Results

An unlimited number of hallucinations from an identity can now be generated by
fixing $z_{1}$ and varying $z_{2}$. A plot showing example generations from
randomly generated identities is shown in Figure 2.

Figure 2: Randomly generated realisations from our model. Column a. shows a
set of five anchor images. Columns b. and c. show two realisations from
varying the pose input for each anchor image with a fixed identity input.
Columns c. and d. show two realisations from varying the identity input and
keeping the pose input fixed.

To generate a dataset for face recognition, a set of 100 realisations are
generated for each of 10,000 different identities. The GAN is trained for 150k
iterations, where-after there is no gain in the quality of images generated.
The dataset is then prepared for face recognition training by aligning and
cropping with MTCNN Zhang et al. (2016) to $112\times 112$ pixels. In
comparison the dataset MS-Celeb-1M contains 6.46 million images from 94,682
identities (Guo et al., 2016) and VGGFace2 contains 3.31 million images from
9,131 identities (Cao et al., 2018).

The dataset is used to train ArcFace using the default settings: a latent
space size of $\mathbbm{R}^{512}$ and an angular margin of 0.5. The training
is terminated after 10 epochs without improvement on the test set of LFW. The
trained model is then evaluated on LFW and achieves a performance of 83.1%
accuracy.

###  5.1 Auxiliary data

To better understand the performance of the synthetic data set, we retrain the
ArcFace face recognition algorithm using the synthetic data supplemented with
varying quantities of real data from the MS-Celeb-1M data set. The 10,000
synthetic identities are combined with a random selection of the 94,682
identities in MS-Celeb-1M and used to train ArcFace using the same parameters
as in the previous experiment.

We find that even a small amount of real data can greatly improve the
performance of our synthetic data set. We observe that the models trained
using real and synthetic data outperform models trained on the same subset of
real data without the synthetic data. With a sufficiently large split we see
that models trained on solely the real data set begin to outperform models
using synthetic and real data. At a larger split this becomes less noticeable
as the size of the real data set begins to dominate the synthetic data set.

Figure 3: Performance of face recognition models trained on varying splits of
real and synthetic data.

###  5.2 Membership attack

We carry out a simple membership attack to show the improved privacy from our
synthetic data set. We assume that we are given black-box access to 1) the
ArcFace model trained on MS-Celeb-1M which was used in training our modified
StyleGAN2 2) the ArcFace model trained on our synthetic data. Given an
identity, the goal of a membership attack is to determine if the identity was
used in the training of the model. Our membership attack seeks to exploit the
difference in prediction entropy of a target model between its training set
and a test set, as per Shokri et al. (2017).

We take one sample from each of the 5749 identities in the LFW data set and an
equal number of distinct identities randomly from MS-Celeb-1M. Both sets of
images are evaluated using ArcFace to produce a feature vector $x$. As in the
training of ArcFace, we calculate the angle $\theta$ between $x$ and the
ground truth weight. We then calculate the logits $\cos(\theta+\theta_{m})$
which are scaled by the feature scale $s$ and softmaxed to probabilities. For
our membership attack, we calculate the entropy of this vector $p$ for each
image by the sum $-\sum_{i}p_{i}\log(p_{i})$. The entropy distributions are
shown in Figure 3 and for each method the images are ranked by their
calculated entropy. We find that with the ArcFace model trained on MS-
Celeb-1M, 78% of the lowest entropy images belong to MS-Celeb-1M, however with
the model trained using our synthetic data set this falls to 51%. Although
this is only a simple attack, it shows that the classifier trained directly on
real data exposes the data more clearly than the one trained on generated
samples.

Figure 4: Distribution of entropies from data sets LFW and MS-Celeb-1M
calculated using networks trained with real and synthetic data.

###  5.3 Biases introduced by the synthetic data set

In this part, we will try to analyse possible explaining factors for the
performance discrepancy between our synthetic data set training of ArcFace and
the real data counterpart. To simplify the notation we will note
$\mathcal{D}_{g}$ our synthetic data set, $\mathcal{D}_{MS}$ the MS-Celeb-1M
data set, $\mathcal{D}_{FF}$ the FFHQ data set, $\mathcal{D}_{LFW}$ the LFW
data set. The bird’s-eye explanation is merely that $\mathcal{D}_{g}$
introduces a larger domain shift with respect to the target validation set
$\mathcal{D}_{LFW}$ than $\mathcal{D}_{MS}$ does. As a result, ArcFace
features are not as robust and informative on $\mathcal{D}_{LFW}$ identities
when trained on $\mathcal{D}_{g}$ than when trained on $\mathcal{D}_{MS}$.

This increased domain shift can be imputed either to i) a bias of
$\mathcal{D}_{g}$ towards $\mathcal{D}_{FF}$ due to the GAN loss (which
involves $\mathcal{D}_{FF}$ directly) or ii) intrinsic biases of the StyleGAN2
network. To evaluate the latter possibility, we also consider a synthetic data
set (denoted by $\mathcal{D}_{gms}$) produced by StyleGAN2 trained in a
classical way but on $\mathcal{D}_{MS}$ instead of $\mathcal{D}_{FF}$.

| $\mathcal{D}_{gms}$ | $\mathcal{D}_{g}$ | $\mathcal{D}_{MS}$ | $\mathcal{D}_{FF}$ | $\mathcal{D}_{LFW}$
---|---|---|---|---|---
$\mathcal{D}_{gms}$ | X | 104.65 | 19.84 | 108.29 | 30.2
$\mathcal{D}_{g}$ | 13.42 | X | 85.10 | 11.51 | 110.24
$\mathcal{D}_{MS}$ | 5.79 | 10.58 | X | 78.85 | 25.01
$\mathcal{D}_{FF}$ | 6.32 | 9.15 | 0.86 | X | 105.58
$\mathcal{D}_{LFW}$ | 6.77 | 11.81 | 1.35 | 2.28 | X
Table 1: Distances between pairs of data sets given in terms of FID (upper
half) and Frechet distance in original ArcFace feature space (lower half).

To evaluate domain shifts, we shall rely on the Frechet distance in two
feature spaces. Indeed we report the Frechet distance between pairs of domains
in the Inception space yielding the FID and in the ArcFace feature space
(trained on $\mathcal{D}_{MS}$). Since distance matrices are symmetric, we
gather the report of both distances in the single table 1. Focusing first on
FID (upper half of the table) and in particular the last column, one observes
that $\mathcal{D}_{LFW}$ is closer to $\mathcal{D}_{MS}$ than to
$\mathcal{D}_{FF}$, which endorses the fact that the domain shift
$\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{FF}$ is larger than
$\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{MS}$. Note that the margin in
terms of FID is severe. The same observation holds for the comparison between
the domain shifts $\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{g}$ and
$\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{gms}$ with again a large
discrepancy. These observations validate our hypothesis i). On the other hand,
to put in balance hypothesis i) and ii), one can only note a slight
discrepancy of FIDs corresponding to
$\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{MS}$ and
$\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{gms}$. This supports the idea
that indeed the SyleGAN2 has an impact in terms of domain shift, but that it
is minor compared to the impact of relying on $\mathcal{D}_{FF}$.

Looking at the distance in the ArcFace feature space (lower half) is also
enlightening because those features are mostly equivariant with respect to
identity factors. In that space, similar observations can be made as in FID by
focusing on the last row (again using $\mathcal{D}_{LFW}$). Indeed, whether we
consider real data sets or synthetic ones, the reliance on $\mathcal{D}_{FF}$
increases the distance to $\mathcal{D}_{LFW}$. However in this feature space,
the discrepancy is less severe. This fact indicates that the domain shift
introduced by the reliance to $\mathcal{D}_{FF}$ concerns mostly other factors
of variation than those tied to identity. This is also confirmed by noting
that the ArcFace Frechet distance corresponding to
$\mathcal{D}_{FF}\leftrightarrow\mathcal{D}_{MS}$ ($0.86$) is smaller than
$\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{MS}$ ($1.35$) while the
comparison was reversed in the more generic space of inception features (FID
of $78.85$ for $\mathcal{D}_{FF}\leftrightarrow\mathcal{D}_{MS}$ vs $25.01$
for $\mathcal{D}_{LFW}\leftrightarrow\mathcal{D}_{MS}$).

The outcome of this analysis is that the decrease of performance in the
downstream task is rather accounted by intrinsic biases of the subsidiary data
set $\mathcal{D}_{FF}$ rather than by the reliance of StyleGAN2 to produce the
synthetic data set. What is more, these biases concern factors of variations
that are uncorrelated with identity, which is the data that we want to
protect. This suggests that gathering a subsidiary data set with less
restricted variations could help fill the gap between our current performance
and the one obtained by training directly on the private data set
$\mathcal{D}_{MS}$.

###  5.4 Conclusion

We have proposed an algorithm for creating synthetic datasets for the use of
training face verification algorithms. It uses changes in the architecture of
the popular StyleGAN2 to enforce the separation of identity and other factors.
We have shown that the GAN generation process does offer an observable
improvement in privacy as demonstrated by a simple membership attack. A
generated dataset which is successful at training a face recognition algorithm
also presents the opportunity to introduce a differentially private mechanism
into the generation process. This avenue is left for future work, as
differential privacy in GANs is not yet mature enough.

Compared to real datasets, our synthetic dataset gives reduced performance on
face recognition tasks, with an accuracy of 83% compared to the state-of-the-
art 99.7%. To investigate the discrepancy in performance we have analysed our
synthetic dataset, comparing it to real datasets across a variety of metrics.
We identity a few key issues. Firstly, the usage of FFHQ in training our GAN
could present a problem as the dataset is not perceptually similar to other
face dataset. In particular, FFHQ is quite restricted in terms of non-identity
related factors compared to "in the wild" type of datasets. As a result, this
lack of intra-class diversity affects our synthetic dataset, which is a key
problem to tackle in the future. Finally, the generated dataset may introduce
difficulties in the downstream task training, due to the inclusion of
generated artifacts which are not perceptually visible but which may still
introduce some domain shift. Our analysis suggests that the latter issue has a
lower impact than the former.

We hope that this work provides some interest in tackling the problem of
privacy in face recognition datasets. As the size and quality of face datasets
increases so does the public’s interest in their personal privacy. Given our
experiments, we endorse that synthetic face datasets provide a promising
solution.

## Acknowledgements

This project was partly funded by the support of the ANR through the BIOQOP
contract referenced ANR-17-CE39-0006.

## References

  * Cao et al. (2018) Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  In _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_, pages 67–74. IEEE, 2018.
  * Choi et al. (2017) Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F Stewart, and Jimeng Sun.  Generating multi-label discrete patient records using generative adversarial networks.  _arXiv preprint arXiv:1703.06490_ , 2017.
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4690–4699, 2019.
  * Donahue et al. (2017) Chris Donahue, Zachary C Lipton, Akshay Balsubramani, and Julian McAuley.  Semantically decomposing the latent spaces of generative adversarial networks.  _arXiv preprint arXiv:1705.07904_ , 2017.
  * Dwork et al. (2014) Cynthia Dwork, Aaron Roth, et al.  The algorithmic foundations of differential privacy.  _Foundations and Trends in Theoretical Computer Science_ , 9(3-4):211–407, 2014.
  * Guo et al. (2016) Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In _European conference on computer vision_ , pages 87–102. Springer, 2016.
  * Henriksen-Bulmer and Jeary (2016) Jane Henriksen-Bulmer and Sheridan Jeary.  Re-identification attacks—a systematic literature review.  _International Journal of Information Management_ , 36(6):1184–1192, 2016.
  * Jordon et al. (2018) James Jordon, Jinsung Yoon, and Mihaela van der Schaar.  Pate-gan: Generating synthetic data with differential privacy guarantees.  In _International Conference on Learning Representations_ , 2018.
  * (9) T Karras.  Flickr-faces-hq dataset (ffhq).
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8110–8119, 2020.
  * Lin et al. (2021) Zinan Lin, Vyas Sekar, and Giulia Fanti.  On the privacy properties of gan-generated samples.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1522–1530. PMLR, 2021.
  * Naker and Greenbaum (2017) Sharon Naker and Dov Greenbaum.  Now you see me: Now you still do: Facial recognition technology and the growing lack of privacy.  _BUJ Sci. & Tech. L._, 23:88, 2017.
  * Odena et al. (2017) Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In _International conference on machine learning_ , pages 2642–2651. PMLR, 2017.
  * Prabhakar et al. (2003) Salil Prabhakar, Sharath Pankanti, and Anil K Jain.  Biometric recognition: Security and privacy concerns.  _IEEE security & privacy_, 1(2):33–42, 2003\.
  * Ruiz et al. (2018) Nataniel Ruiz, Eunji Chong, and James M Rehg.  Fine-grained head pose estimation without keypoints.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 2074–2083, 2018.
  * Saxena and Cao (2021) Divya Saxena and Jiannong Cao.  Generative adversarial networks (gans) challenges, solutions, and future directions.  _ACM Computing Surveys (CSUR)_ , 54(3):1–42, 2021\.
  * Senior and Pankanti (2011) Andrew W Senior and Sharathchandra Pankanti.  Privacy protection and face recognition.  In _Handbook of face recognition_ , pages 671–691. Springer, 2011\.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In _2017 IEEE Symposium on Security and Privacy (SP)_ , pages 3–18. IEEE, 2017.
  * Trigueros et al. (2018) Daniel Sáez Trigueros, Li Meng, and Margaret Hartnett.  Generating photo-realistic training data to improve face recognition accuracy.  _arXiv preprint arXiv:1811.00112_ , 2018.
  * Zhang et al. (2016) K. Zhang, Z. Zhang, Z. Li, and Y. Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  _IEEE Signal Processing Letters_ , 23(10):1499–1503, Oct 2016.  ISSN 1070-9908.  10.1109/LSP.2016.2603342.