# Evaluating Transferability for Covid 3D Localization Using CT SARS-CoV-2
segmentation models

Constantine Maganaris  National Technical University of Athens, School of
Rural, Surveying and Geoinformatics EngineeringAthensGreece<EMAIL_ADDRESS>0000-0003-2738-684X ,  Eftychios Protopapadakis  National Technical University
of Athens, School of Rural, Surveying and Geoinformatics
EngineeringAthensGreece<EMAIL_ADDRESS>,  Nikolaos Bakalos  Institute of
Communications and Computer SystemsAthensGreece<EMAIL_ADDRESS>,
Nikolaos Doulamis  Institute of Communications and Computer
SystemsAthensGreece<EMAIL_ADDRESS>,  Dimitris Kalogeras  Institute of
Communications and Computer SystemsAthensGreece<EMAIL_ADDRESS>and
Aikaterini Angeli  Institute of Communications and Computer
SystemsAthensGreece<EMAIL_ADDRESS>
###### Abstract.

Recent studies indicate that detecting radiographic patterns on CT scans can
yield high sensitivity and specificity for Covid-19 localization. In this
paper, we investigate the appropriateness of deep learning models
transferability, for semantic segmentation of pneumonia-infected areas in CT
images. Transfer learning allows for the fast initialization/reutilization of
detection models, given that large volumes of training data are not available.
Our work explores the efficacy of using pre-trained U-Net architectures, on a
specific CT data set, for identifying Covid-19 side-effects over images from
different datasets. Experimental results indicate improvement in the
segmentation accuracy of identifying Covid-19 infected regions.

datasets, neural networks, segmentation, unet, Covid, ct

††copyright: none††ccs: Applied computing Health informatics

##  1\. Introduction

Coronavirus pandemic (COVID-19) and its variations, has infected more than 427
million people and caused more than 5,9 million deaths around the globe, based
on the facts of John Hopkins University (22 February 2022)
(noauthor_Covid-19_nodate, 1). Since the declaration of Public Health
emergency of International Concern on January 30 2020
(noauthor_statement_nodate, 2) from the World Health Organization (WHO) there
was a research outbreak in the field, to respond the emergency situation. A
great amount of research has been focused since then on fast detection of
COVID, using CT or X-rays of the thorax, in parallel with other methods like
Antigenic and Reverse Transcription-Polymerase Chain Reaction (RT-PCR)
testing.

Currently, the WHO recommends the use of rapid tests (noauthor_antigen-
detection_nodate, 3) in general population for primary case detection in
symptomatic individuals suspected to be infected and asymptomatic individuals
at high risk of COVID-19, for contact tracing, during outbreak investigations
and to monitor trends of disease incidence in communities, even if they have
poor sensitivity in their results (ferte_accuracy_2021, 4). The RT-PCR tests
are also known to have relatively high false negative rates (kortela_real-
life_2021, 5). This made crucial to invent methodologies to detect fast and
reliably COVID infection but also to estimate the cruciality of it or how fast
it expands in a patient, to curate it better, these data are not available
with the methods of PCR or Antigen testing. This makes the use of CT and X-ray
scans a mandatory tool in the hands of clinicians to work with COVID patients,
but also suggests the need for creating new tools using Artificial
Intelligence to assist the experts in their work and make detection of the
effects of COVID faster.

It was 2016, at a Conference in Toronto, when Geoffrey Hinton said that “[…]
People should stop training radiologists now, it is just completely obvious
that in 5 years Deep learning is going to do better than radiologists because
this can be able to hit a lot more experience. It might be 10 years but we got
plenty of radiologists already” (creative_destruction_lab_geoff_2016, 6).
Nevertheless, things progressed differently. During the pandemic, it became
clear that the AI tools cannot replace radiologists, which were much in need
and the pandemic had a great impact in them, especially in cases like Northern
Italy (coppola_impact_2021, 7).

Despite all that, it is crucial to continue developing automated decision
making tools to assist healthcare personnel and overcome all the issues that
comes with the analysis of Computed Tomography Imaging. One of the main
setbacks in creating accurate models is the lack of publicly available
datasets with chest scans of infected people. In addition, a lot of data that
have been used so far, have faulty cleansing, so they create frustration on
the results produced by deep learning models, trained to detect COVID
(noauthor_hundreds_nodate, 8). Moreover, scans coming from different
equipment, produce data that differentiate on how they signal COVID areas
using the Hounsfield scale thus making the training of appropriate models even
more difficult due to the significant differences in the training data.

In our research we use two different datasets in order to detect the
transferability of deep learning models between them, and estimate the
accuracy of them to detect segments of COVID areas (i.e. markings of ground
glass opacities, consolidation and pleural) (cozzi_ground-glass_2021, 9).
These datasets include 3410 slides of CT scans with annotated COVID and Lung
segments, but also have masks of the aforementioned signals of COVID-19.

From our results we have found that there is evidence of such transferability
of results, after retraining the model in a portion of the data of a second
dataset and for a short number of epochs for the second train. It is important
to note that we achieved a great prediction outcome with good rankings in
precision, recall and F1 metrics. To present our results, we have visualized
3D reconstructions of the lungs using data from the CT scans. This way we
achieved better understanding on how the model works in the complete CT scan
and not by comparing separate slides of it. In our 3D model reproduction, we
used only the lung areas of the slides, the annotations of the COVID areas
made by the radiologists (ground truth) and the predicted COVID areas from our
model. Now having these representations in hand we can view the bigger picture
and how the model performs in a patient. This is making it easier for a user
(i.e. a doctor) of the model to extract information from the results of how
serious the illness is, or how the infection responds to medication etc. by
comparing different scans of the same patience in different points of time.

Another finding of our research was that even in our set of data, there were
COVID areas (GGOs) falsely not annotated by the radiologists, we have seen
that our model managed to correctly annotate COVID affected areas.
Additionally, areas that have been falsely marked as COVID by the
radiologists, and the model has been trained in them, produces correct output
(i.e. non Covid) from our model.

##  2\. Related work

Deep learning methodologies using various types of images are common for
identfication, detection or segmentation in medical imaging
(voulodimos_deep_2018, 10) and in biomedical applications (le_fertility-
gru_2019, 11). In this context, researchers already investigating several
approaches to assist medical professionals with Covid-19 detection. An initial
approach was to classify multiple CT slices using a convolutional neural
network variation (li_coronavirus_2020, 12). The adopted methodology is able
to identify a viral infection with a ROCAUC score of 0.95 (a score of 1
indicates a perfect classifier). However, despite the high detection rates,
the authors indicated that it was extremely difficult to distinguish among
different types of viral pneumonia based solely on CT analysis.

Convolutional Neural Network (CNN) variations for the distinction of
coronavirus vs. non-coronavirus cases have been proposed by (li_using_2020,
13). The specific approach allows for a distinction among Covid-19, other
types of viral infections, and non-infection cases. Results indicate that
there are adequate detection rates and a higher detection rate than RT-PCR
testing. Towards this direction, CNN structures are combined with Long Short
Term Memory (LSTM) networks to improve the classffication accuracy of CNN
networks further (islam_combined_2020, 14). Additionally the work of (fan_inf-
net_2020, 15) introduced a parallel partial decoder, called Inf-Net, which
combines aggregation of high-level features to generate a global map. This is
achieved through the use of convolutional hierarchies.

A U-Net-based model, named U-Net++, was applied to high-resolution CT images
for Covid-19 detection in (chen_deep_2020, 16). Furthermore, in
(ardakani_application_2020, 17) a system for the detection of Covid-19 using
10 variants of CNNs in CT images is proposed, including AlexNet, VGG-16,
VGG-19, SqueezeNet, GoogleNet, MobileNet-V2, ResNet-18, ResNet-50, ResNet-101,
and Xception. ResNet-101 and Xception outperformed the remaining ones. AlexNet
and Inception-V4 were also used for Covid-19 detection in CT scans in
(cifci_deep_2020, 18). The framework presented in (singh_classification_2020,
19) used a CNN and an Artificial Neural Network Fuzzy Inference System
(ANNFIS) to detect Covid-19, whereas a Stack Hybrid Classification (SHC)
scheme based on ensemble learning is proposed in (farid_novel_2017, 20).
Focusing on segmentation, a type 2 fuzzy clustering system combined with a
Super-pixel based Fuzzy Modified Flower Pollination Algorithm is proposed in
(chakraborty_sufmofpa_2021, 21) for Covid-19 CT image segmentation.

Finally in (katsamenis_transfer_2020, 22) the experimental results indicate
that the transfer learning approach outperforms the performance obtained
without transfer learning, for the Covid-19 classification task in chest X-ray
images, using deep classification models, such as convolutional neural
networks (CNNs).

##  3\. Experimental setup

The U-Net and the data transformation scripts, were developed in Python 3
using TensorFlow and Keras libraries. The models were trained in a VM using
Unix MATE OS with 8 Core CPU and 64GB RAM provided GRNet Synefo service.
Figure 1 presents the architecture of the U-Net model.

Figure 1. U-Net Architecture Table 1. Datasets’ properties Set Name |  | # DICOM Files
---
# Slides |  | Slides with
---
lung areas
| Slides with
---
Covid areas
| Train set slides
---
(ratio 50%/50%
Covid/non-Covid slides)
| Validation set slides
---
(ratio 50%/50%
Covid/non-Covid slides)
| Test set
---
Slides
Epochs
CT 1-9 | 9 | 829 | 713 | 373 | 440 | 60 | 214 | 111
CT 10 | 10 | 2581 | 2156 | 1351 | 440 | 60 | 1656 | 46

###  3.1. Datasets description

To extract our results, we used two Lung Covid infected datasets, the Covid-19
CT segmentation dataset (noauthor_Covid-19_nodate-1, 23) from which we used
only the Segmentation dataset nr. 2, this includes 9 DICOM files of continuous
lung CT scans and the 20th April update (jun_Covid-19_2020, 24) which contains
another 20 labeled Covid-19 CT scans, from this we used only the 10 files
marked as Coronacases and not the Redeopedia ones. The reason for this, is
because all DICOM files we used, contained data in the Hounsfield scale
(PMID:31613501, 25) the Radeopedia set of DICOM files, contained pixel values
in the range of 0-255 therefore we could not use it since it did not follow
our normalization procedure. The first set of 9 DICOM files (we refer to this
set as CT 1-9), contained 829 slides of CT, having dimensions of 630x630
pixels and includes already hand annotated lung and Covid masks for each slide
of it. Similarly, the Coronacases dataset contained 10 CT scans (we refer to
this set as CT 10) with 2581 slides in total having dimensions of 512x512 and
includes also annotated masks by radiologists of the lung and Covid areas.
Both sets, include continuous slides of complete lung CT scans of the same
patient and not slides of different patients in each DICOM file. To construct
our dataset we used only the slides that include lung areas, in order to
achieve better results by reducing the extra information of slides without
lung areas. But looking at Table LABEL:table:1, we see that the data in the CT
1-9 dataset use 214 train + 60 validation + 440 test slides total of 714
slides, but we see that the slides with lung areas are in total 713. This
happens since one of the slides, marks a tiny area of few pixels as Covid,
even if there is no lung or Covid in it, like in Figure 6 (b). This is due to
a human error in the annotation procedure. For the normalization process, we
resized the images to 320x320 pixels using Nearest Neighbor Interpolation and
we kept only the Hounsfield values in the range of -970 to -150. All the
information that is needed in our paradigm of Covid segmentation in lung
areas, relies in this spectrum of Hounsfield scale. To achieve this, we
normalize each pixel based on the following type

(1) $\frac{(pixelVal-minHounsfield)}{(maxHounsfield-
minHounsfield)}=\frac{(pixelVal+970)}{(-150+970)}$

For every pixel value we get that is greater than 1. we assign the value 1 and
every value less than 0. we assign the value 0. This way our dataset includes
only values in the range [0.,1.] The radiologists have also marked in separate
files the lung masks and the Covid masks for each slide of the CT scans we
used. Therefore we arranged our Training Input in the form of n X 320 X 320 X
2 since in the first channel we used the normalized values of the CT slides
and in the second channel we used the lung masks of each slide using binary
values of 0,1 (1 in pixels that are marked as lung, 0 elsewhere). The output
was having the form of n X 320 X 320 X 1 signaled with binary values 0,1 (1 in
pixels that are marked as Covid, 0 elsewhere).

###  3.2. Performance metrics

In order to objectively evaluate our results, four different metrics are
considered: accuracy, precision, recall, and the F1‐score, which is directly
calculated from precision and recall values. Accuracy (ACC) is defined as:

(2) ${ACC}=\frac{(TP+TN)}{(TP+TN+FP+FN)}$

where the nominator contains the true positives (TP) and true negatives (TN)
samples, while denominator contains the TP and TN and false positives (FP) and
false negatives (FN). Precision, recall and F1‐score are given as:

(3) ${Pr}=\frac{TP}{(TP+FP)}$

(4) ${Re}=\frac{TP}{(TP+FN)}$

(5) ${F1}=\frac{Pr\times Re}{(Pr+Re)}$

###  3.3. Experimental results

With the pre-described formation of both datasets, we started the training of
a Unet using the CT 1-9 data, with 4 encoding/decoding layers having as input
440 x 320 x 320 x 2 were in the 1st channel we have the data of the CT scan
and in the 2nd channel we have the lung mask of the specific CT slide. This
way the model focuses only in the lung areas of the CT scans in the learning
process. In the output we have 440 x 320 x 320 x 1 where in the only channel
we have the masks data of the Covid areas. We also used a validation set of 60
x 320 x 320 x 2 as input and 60 x 320 x 320 x 1 as output. In the Unet we used
(i.e. Figure 1) the rectified linear activation unit (ReLU) function for the
3x3 conv layers and the Sigmoid activation in the 1x1 conv layer, to get
output values in the range (0,1), a learning rate of 0.0001, a batch size of
45 and the shuffling enabled. The max epochs were 200 from which it only used
111 till the early stopping engaged.

We then extracted metrics for the test data of the CT 1-9 set (i.e. 214
slides) these were F1 score, Accuracy, Precision and Recall (see Table 2),
from which we got a great Accuracy value with an average of 0.9973 and an F1
score with an average of 0.7832 in total.

Table 2. Performance Metrics | Acc | Pre | Rec | F1
---|---|---|---|---
CT 1-9 | 0.997375 | 0.751451 | 0.833533 | 0.783245
CT 10 (no retrain) | 0.995027 | 0.628606 | 0.647012 | 0.613689
CT 10 (retrain) | 0.997397 | 0.813515 | 0.863668 | 0.827899

In Figure 3 we can see that the model performs great in the test set of the
1st dataset, even in a bad scenario where the F1 Score is 0.4164 for this
specific image.

Figure 2. 1st Dataset Good Prediction Example; CT scan (left), Ground Truth
(middle), Prediction (right)

Figure 3. 1st Dataset Bad Prediction Example; CT scan (left), Ground Truth
(middle), Prediction (right)

Afterwards we moved with the extraction of the same metrics, using all slides
of set of the second dataset (CT 10). From these we got an average accuracy of
0.995 but an F1 score of 0.6137. Due to the low F1 score, we re-trained our
model using a small portion of this dataset. From CD 10 dataset we had a
similar size of train and validation sets (440 and 60) and we continued the
train of the previous model but now we reduced the max epochs to 50. From
these epochs the train process used 46 till the early stopping engaged. We
again used the re-trained model in the test set of the 2nd dataset which
includes 1656 slides to obtain our metrics.

Figure 4. 2nd Dataset Good Prediction Example; CT scan (left), Ground Truth
(middle), Prediction (right)

We must note here that all the slides that have been used from both datasets,
had an area marked as lung in the accompanied mask files from the
radiologists. If a slide did not include a lung area, it was not included in
the any of the trained/validation or test set data. From the extraction of
metrics in the retrained model, we got an average accuracy of 0.9974 and an
average F1 score of 0.8279 which was an improvement compared to the 0.6137 F1
before the retrain of the model.

In Figure 5 see an example of how the re-trained model performs, even in the
bad scenario where the F1 Score of is 0.0136 we cannot clearly see an actual
GGO on the CT image, even if in the Covid masks the radiologists marks one.

Figure 5. 2nd Dataset Bad Prediction Example; CT scan (left), Ground Truth
(middle), Prediction (right)

##  4\. Dataset limitations

In the datasets we have used, we spotted some inconsistencies, since its
annotation was made by hand. Specifically in our pre-train analysis, we found
that in DICOM files there were slides with a false marked Covid areas. As we
can see in Figure 6 (b) the radiologist has a marked Covid area in a section
that there is no lung at all.

Figure 6. False Covid annotations; Area 1 (left), Area 2 (right)

We have also spotted a case where our model predicted a Covid area even if it
was not marked as one by the radiologist. From the Figure 7 we can see that
there is a GGO in the CT scan area, predicted by our model, even if there is
no mask in ground truth.

Figure 7. Missing annotated Covid areas, in the original dataset; CT scan
(left), Ground Truth (middle), Prediction (right)

Taking account the previously mentioned paradigms, we cannot be sure of the
extent of these faulty annotated areas, since the only chance to find them is
to re-evaluate all annotations by a different radiologist and compare the
results.

##  5\. 3D representations

To assist the medical personnel which works with tools like Computed
Tomography scans, we constructed a python script that exports a 3D
representation of the CT scan data and the Covid segments produced by our
model. At first, we extracted for every slide in a CT the color value of each
pixel. We created an array of color values only in the lung areas, each entry
of the array was having the form of (x position, y position, z position, color
value) The z position was computed starting from 0 and adding a constant value
for every new slide that we export its data. We did the same procedure for the
Covid masks that our model predicted and for the ground truth, but in these
the color value was in binary (i.e. 0 or 1). We then exported these arrays in
comma separated files which we then imported to the ParaView visualizer
(noauthor_paraview_nodate, 26) using the table to point filtering.

Figure 8. 3D Representation; CT scan (left), Ground Truth (middle), Prediction
(right)

With the 3D reconstruction, the evaluation of a patience status is easier for
the medical personnel, as they can see the whole lung and not evaluate
separate CT slides which can be significant in number. With this tool it is
also easier to estimate how a treatment performs in time, using different 3D
representations of the same patient. From the Fig 8 we can compare the model
prediction with the Ground Truth but we can also see that in the Ground Truth,
there are faulty annotated Covid areas outside the lung segments (marked in
the Ground Truth), made by the radiologists.

###### Acknowledgements.

This research has been co‐financed by European Union’s Horizon 2020 research
and innovation programme under grant agreement No 883441 for the STAMINA
Innovation action.

## References

  * (1) “COVID-19 Map”  In _Johns Hopkins Coronavirus Resource Center_ , 2022  URL: https://coronavirus.jhu.edu/map.html
  * (2) “Statement on the tenth meeting of the International Health Regulations (2005) Emergency Committee regarding the coronavirus disease (COVID-19) pandemic”, 2022  URL: https://www.who.int/news/item/19-01-2022-statement-on-the-tenth-meeting-of-the-international-health-regulations-(2005)-emergency-committee-regarding-the-coronavirus-disease-(covid-19)-pandemic
  * (3) “Antigen-detection in the diagnosis of SARS-CoV-2 infection”, 2022  URL: https://www.who.int/publications-detail-redirect/antigen-detection-in-the-diagnosis-of-sars-cov-2infection-using-rapid-immunoassays
  * (4) Thomas Ferté et al.  “Accuracy of COVID-19 rapid antigenic tests compared to RT-PCR in a student population: The StudyCov study”  In _Journal of Clinical Virology: The Official Publication of the Pan American Society for Clinical Virology_ 141, 2021, pp. 104878  DOI: 10.1016/j.jcv.2021.104878
  * (5) Elisa Kortela et al.  “Real-life clinical sensitivity of SARS-CoV-2 RT-PCR test in symptomatic patients”  In _PloS One_ 16.5, 2021, pp. e0251661  DOI: 10.1371/journal.pone.0251661
  * (6) Creative Destruction Lab  “Geoff Hinton: On Radiology”, 2016  URL: https://www.youtube.com/watch?v=2HMPRXstSvQ
  * (7) Francesca Coppola et al.  “Impact of the COVID-19 outbreak on the profession and psychological wellbeing of radiologists: a nationwide online survey”  In _Insights into Imaging_ 12.1, 2021, pp. 23  DOI: 10.1186/s13244-021-00962-2
  * (8) “Hundreds of AI tools have been built to catch covid. None of them helped.”  In _MIT Technology Review_ , 2022  URL: https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/
  * (9) Diletta Cozzi et al.  “Ground-glass opacity (GGO): a review of the differential diagnosis in the era of COVID-19”  In _Japanese Journal of Radiology_ 39.8, 2021, pp. 721–732  DOI: 10.1007/s11604-021-01120-w
  * (10) Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis and Eftychios Protopapadakis  “Deep Learning for Computer Vision: A Brief Review” Publisher: Hindawi  In _Computational Intelligence and Neuroscience_ 2018, 2018, pp. e7068349  DOI: 10.1155/2018/7068349
  * (11) Nguyen Quoc Khanh Le  “Fertility-GRU: Identifying Fertility-Related Proteins by Incorporating Deep-Gated Recurrent Units and Original Position-Specific Scoring Matrix Profiles” Publisher: American Chemical Society  In _Journal of Proteome Research_ 18.9, 2019, pp. 3503–3511  DOI: 10.1021/acs.jproteome.9b00411
  * (12) Yan Li and Liming Xia  “Coronavirus Disease 2019 (COVID-19): Role of Chest CT in Diagnosis and Management” Publisher: American Roentgen Ray Society  In _American Journal of Roentgenology_ 214.6, 2020, pp. 1280–1286  DOI: 10.2214/AJR.20.22954
  * (13) Lin Li et al.  “Using Artificial Intelligence to Detect COVID-19 and Community-acquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy” Publisher: Radiological Society of North America  In _Radiology_ 296.2, 2020, pp. E65–E71  DOI: 10.1148/radiol.2020200905
  * (14) Md. Zabirul Islam, Md. Milon Islam and Amanullah Asraf  “A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images”  In _Informatics in Medicine Unlocked_ 20, 2020, pp. 100412  DOI: 10.1016/j.imu.2020.100412
  * (15) Deng-Ping Fan et al.  “Inf-Net: Automatic COVID-19 Lung Infection Segmentation From CT Images” Conference Name: IEEE Transactions on Medical Imaging  In _IEEE Transactions on Medical Imaging_ 39.8, 2020, pp. 2626–2637  DOI: 10.1109/TMI.2020.2996645
  * (16) Jun Chen et al.  “Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography” Number: 1 Publisher: Nature Publishing Group  In _Scientific Reports_ 10.1, 2020, pp. 19196  DOI: 10.1038/s41598-020-76282-0
  * (17) Ali Abbasian Ardakani et al.  “Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: Results of 10 convolutional neural networks”  In _Computers in Biology and Medicine_ 121, 2020, pp. 103795  DOI: 10.1016/j.compbiomed.2020.103795
  * (18) Mehmet Cifci  “Deep Learning Model for Diagnosis of Corona Virus Disease from CT Images”  In _International Journal of Scientific Research and Management_ 11, 2020, pp. 273
  * (19) Dilbag Singh, Vijay Kumar, Vaishali and Manjit Kaur  “Classification of COVID-19 patients from chest CT images using multi-objective differential evolution–based convolutional neural networks”  In _European Journal of Clinical Microbiology & Infectious Diseases_ 39.7, 2020, pp. 1379–1389  DOI: 10.1007/s10096-020-03901-z
  * (20) Ahmed Abdullah Farid, Gamal Ibrahim Selim and Hatem Awad A Khater  “A Novel Approach of CT Images Feature Analysis and Prediction to Screen for Corona Virus Disease (COVID-19)”  In _International Journal of Scientific and Engineering Research_ 11.3, 2017, pp. 9
  * (21) Shouvik Chakraborty and Kalyani Mali  “SuFMoFPA: A superpixel and meta-heuristic based fuzzy image segmentation approach to explicate COVID-19 radiological images”  In _Expert Systems with Applications_ 167, 2021, pp. 114142  DOI: 10.1016/j.eswa.2020.114142
  * (22) Iason Katsamenis et al.  “Transfer Learning for COVID-19 Pneumonia Detection and Classification in Chest X-ray Images” Type: article, 2020, pp. 2020.12.14.20248158  DOI: 10.1101/2020.12.14.20248158
  * (23) “COVID-19”  In _Medical segmentation_ , 2022  URL: http://medicalsegmentation.com/covid19/
  * (24) Ma Jun et al.  “COVID-19 CT Lung and Infection Segmentation Dataset” Type: dataset  Zenodo, 2020  DOI: 10.5281/zenodo.3757476
  * (25) Tami D. DenOtter and Johanna Schubert  “Hounsfield Unit”  StatPearls Publishing, Treasure Island (FL), 2021  URL: http://europepmc.org/books/NBK547721
  * (26) “ParaView”, 2022  URL: https://www.paraview.org/


# Quantum generative adversarial learning for simultaneous multiparameter
estimation

Zichao Huang  Department of Physics and Collaborative Innovation Center for
Optoelectronic Semiconductors and Efficient Devices, Xiamen University, Xiamen
361005, China  Yuanyuan Chen<EMAIL_ADDRESS>Department of Physics and
Collaborative Innovation Center for Optoelectronic Semiconductors and
Efficient Devices, Xiamen University, Xiamen 361005, China  Lixiang Chen
<EMAIL_ADDRESS>Department of Physics and Collaborative Innovation Center
for Optoelectronic Semiconductors and Efficient Devices, Xiamen University,
Xiamen 361005, China

###### Abstract

Generative adversarial learning is currently one of the most prolific fields
in artificial intelligence due to its great performance in a variety of
challenging tasks such as photorealistic image and video generation. While a
quantum version of generative adversarial learning has emerged that promises
exponential advantages over its classical counterpart, its experimental
implementation and potential applications with accessible quantum technologies
remain explored little. Here, we report an experimental demonstration of
quantum generative adversarial learning with the assistance of adaptive
feedback that is based on stochastic gradient descent algorithm. Its
performance is explored by applying this technique to the adaptive
characterization of quantum dynamics and simultaneous estimation of multiple
phases. These results indicate the intriguing advantages of quantum generative
adversarial learning even in the presence of deleterious noise, and pave the
way towards quantum-enhanced information processing applications.

##  I Introduction

Generative adversarial networks (GAN) has been widely used for image and video
processing, pattern recognition, secure steganography and molecule development
salimans2016improved; wang2018multiscale; ledig2017photo. In GAN algorithm, a
generator tries to optimize her strategy over a number of trials, and generate
statistics of data to make a discriminator unable to discriminate between the
generated data and real data. Thus, the generator and the discriminator can be
thought to be adversaries in a machine learning game. The endpoint of such an
adversarial game is the unique Nash equilibrium, indicating that the generator
finds the correct statistics of data and the discriminator unable to tell the
difference heusel2017gans. Although GAN works well in training the generator
to learn the statistics of true data and leads to significant interest in
industries, the requirement of huge computing power makes them to be big
challenges for classical computer that is limited by Moore’s law. In
particular, with the rapid growth of data in volume and dimensionality, the
classical GAN model is getting exponentially larger and require greatly
increased computational resource sagiroglu2013big; marx2013the. To deal with
these complicated learning tasks, quantum version of GAN (QGAN) has aroused
widespread research interest both in theory and experiment
biamonte2017quantum; lloyd2018quantum; dallaire-demers2018quantum;
zoufal2019quantum; hu2019quantum. As quantum information processing can
exhibit an exponential advantage over classical counterparts, QGAN has the
potential to provide speed-up over the classical algorithms he2016deep;
bishop2006pattern; duchi2011adaptive; goodfellow2014generative. However, the
experimental implementation of QGAN with accessible quantum resources in
current stage remains explored little.

Here, we experimentally demonstrate a quantum version of generative
adversarial network (QGAN) with the assistance of adaptive feedback that is
based on stochastic gradient descent algorithm (namely the self-guided QGAN),
wherein both the processing datasets and the projective measurement are
quantum from the beginning. Self-guided QGAN can work with any multilevel
systems, here we use photonic qubits because they are conveniently prepared
and measured at room temperature, which can also be readily extended to a
quantum system with higher dimensions zoufal2019quantum; hu2019quantum. In
order to evaluate its performance, we apply this self-guided QGAN to the
adaptive characterization of quantum dynamics (ACQD). While the conventional
characterization methods face notorious challenges in performing the full
Bell-state measurement (BSM) mohseni2006direct; mohseni2007direct;
graham2013hyperentanglement; wang2007experimental or requiring a large number
of ensemble measurements kim2020universal; luchnikov2020machine;
shabani2011efficient, our QGAN algorithm uses Hong-Ou-Mandel (HOM)
interference to obtain the distinguishability in a single-shot measurement
that eludes the requirement of BSM hong1987measurement; nagali2009optimal, and
this distinguishability works as a realtime feedback for self-guided learning
that decreases the number of experimental trials. Additionally, we also
explore the application of self-guided QGAN in the simultaneous estimation of
multiple phases, which is a crucial tool for quantum-enhanced sensing and
imaging, and may more complex quantum computation and quantum communication
protocols gorecki2022multiple; liu2021distributed; humphreys2013quantum;
hong2021quantum; pezze2017optimal.

While machine learning has been widely used for enhancing the performance of
quantum state tomography (QST)prl2016experiment; rambach2021robust;
ferrie2014self; hou2020experimental, we identify four main advantages
associated with our QGAN method using HOM interference. (i) It is well known
that the projective measurement is still a technical challenge for frequency-
encoded and time-encoded quantum states since they have the requirements of
inefficient frequency shifter guo2017testing or complicated Fresnel
interferometer bulla2022non; vedovato2018postselection;
marcikic2004distribution. Conversely, HOM interference can measure the
distinguishability in any degree of freedom, thus our QGAN method is available
in a wider range of applications. (ii) Since HOM interference can reveal the
information of incident entangled state, our QGAN method has the potential to
characterize the quantum entanglement accordingly. (iii) HOM interference is
determined by the distinguishable information in all degree of freedom, our
QGAN method can learn the quantum state encoded in multiple degrees of freedom
simultaneously, which is an essential tool for hyperentanglement-based quantum
information processing and quantum microscopy ndagano2022quantum. (iv) As a
direct result of inherent stability of HOM interference measurement, our QGAN
approach promises great robustness against deleterious noise and experimental
imperfection, which has the potential to be widely used in the future
practical applications hong1987measurement.

Thus this scheme may provide an alternative approach for fundamental test of
quantum mechanics, and shedding the light on more quantum information
processing tasks that are significantly improved by quantum machine learning
algorithms xia2018quantum; schuld2019quantum; cai2015entanglement.

Figure 1: Schematic of self-guided quantum generative adversarial network,
wherein the quantum states are depicted on the Bloch spheres. (a) The quantum
true state described by $\lvert\psi\rangle$ is provided by a quantum system
whose internal physical structure and quantum process is not required to be
known. (b) The discriminator plays a role in distinguishing the quantum true
state $\lvert\psi\rangle$ and fake state $\lvert\phi\rangle$, which is shown
as green and red Bloch vector respectively. (c) The generator takes the
discriminator’s measurement outcome as a real-time feedback to update the
latent variables of fake state by using stochastic gradient descent algorithm.

##  II Self-guided QGAN by using quantum interference

Figure 1 shows the schematic of self-guided QGAN method. The quantum true data
described by $\lvert\psi\rangle$ is provided by a quantum system, whose
internal physical structure and quantum process is not required to be known.
The generator can prepare arbitrary quantum data that is described by
$\lvert\phi\rangle$, whose purpose is to mimic the quantum true data with
certain probability. The discriminator $M$ performs joint quantum measurement
on the quantum true data and the generated fake data, and attempts to
distinguish between them by analyzing the statistics of measurement results.
In the self-guided QGAN, the generator can receive the information about the
measurement outcomes that works as a real-time feedback, and then adaptively
adjusts his strategies alternatively to fool the discriminator, until the
discriminator cannot distinguish the difference between the true data and fake
data any more. Thus, the generator learns the statistics of the quantum true
data with great advantages in precision and accuracy.

Compared to other QGAN methods, we solve the optimization problem in the
process of the generator by using stochastic gradient descent algorithms
rambach2021robust; ferrie2014self. It finds the correct quantum true data by
iteratively maximizing the overlap between the quantum true data and the
generated fake data using two quantum interference measurements per iteration
independently. In each iteration, the algorithm measures the similarity of the
quantum true data and two random fake data as
$f(\lvert\phi_{\pm}\rangle)=|\langle\phi_{\pm}|\psi\rangle|^{2}$ to
approximate a gradient which heads to the true data more rapidly and then
update the generated fake data. Here, the quantum data are encoded into two
coherent single photons, which are routed into different input ports of the
beam splitter in a HOM interferometer simultaneously hong1987measurement;
chen2018polarization. Thus the bunching probability is directly related to the
photons’ level of indistinguishability, namely the overlap of two incident
photons as described by $|\langle\phi_{\pm}|\psi\rangle|^{2}$.

In a specific iteration $k$, the algorithm chooses one random direction as
$\Delta_{k}\in\\{1,-1,i,-i\\}$, and generates two projection states as
$\lvert\phi_{\pm}\rangle=\lvert\phi_{k}\pm\beta_{k}\Delta_{k}\rangle$.
Thereinto, $\beta_{k}=b/(k+1)^{t}$, where $(b,t)$ are hyperparameters for
controlling the gradient estimation step size. The system then calculates a
gradient $g_{k}$based on the similarity measurement results as

$g_{k}=\frac{f(\lvert\phi_{+}\rangle)-f(\lvert\phi_{-}\rangle)}{2\beta_{k}}\Delta_{k}.$
(1)

Since the visibility of HOM interference can reveal the indistinguishable
level between two measured photons, it can be used directly in Eq. 1 by
replacing as

$\begin{split}f(\lvert\phi_{+}\rangle)-f(\lvert\phi_{-}\rangle)&=f(\lvert\phi_{k}+\beta_{k}\Delta_{k}\rangle)-f(\lvert\phi_{k}-\beta_{k}\Delta_{k}\rangle)\\\
&\propto\frac{|\langle\phi_{k}+\beta_{k}\Delta_{k}|\psi\rangle|^{2}-|\langle\phi_{k}-\beta_{k}\Delta_{k}|\psi\rangle|^{2}}{|\langle\phi_{k}+\beta_{k}\Delta_{k}|\psi\rangle|^{2}+|\langle\phi_{k}-\beta_{k}\Delta_{k}|\psi\rangle|^{2}}\\\
&\propto V_{+}-V_{-},\end{split}$ (2)

where $V_{\pm}$ denote the visibilities of HOM interference from the overlap
measurements between the projection states $\lvert\phi_{\pm}\rangle$ and the
quantum true state $\lvert\psi\rangle$ for each iteration of self-guided QGAN.
Since we are interested in the gradient direction in which we get higher
overlap probability rather than the actual values of the coincidence counts,
the interference visibilities are all normalized for the sake of simplicity.
Then, the generated state is updated by following

$\lvert\phi_{k+1}\rangle=\lvert\phi_{k}+\alpha_{k}g_{k}\rangle,$ (3)

where $\alpha_{k}=a/(k+1+A)^{s}$, $a$, $A$, and $s$ are hyperparameters for
controlling the convergence step size. This procedure is repeated for each
iteration by updating
$\lvert\phi_{k}\rangle\rightarrow\lvert\phi_{k+1}\rangle$ until the
discriminator cannot distinguish the difference between the ture state and
fake state. As shown in the inset of Fig. 1(c), the self-guided QGAN is
steered by the gradient $g_{k}$ and will thus converge to the underlying state
after a sufficient number of iterations. The values of these hyperparameters
can depend on the specific system that is under investigation. While it is
possible to arbitrarily optimize the fidelity by increasing the number of
iterations, it can be chosen to match the required accuracy and iterative
speed that typically depends on the practical applications.

##  III Self-guided QGAN for adaptive characterization of quantum dynamics

Figure 2: Experimental implementation of self-guided QGAN. POL: polarizor;
HWP: half-wave plate; QWP: quarter-wave plate; TEC: temperature controller;
PPKTP: periodically poled potassium titanyl phosphate nonlinear crystal; LP:
longpass filter; PBS: polarizing beam splitter; BS: beam splitter; SPDM:
single photon detection module; $\lvert\phi\rangle$: quantum fake state;
$\lvert\psi\rangle$: quantum ture state. As shown in the inset, the visibility
of the well-known HOM dip will gradually drop to the minimum value with the
self-guided algorithm, and the measured results of HOM interference work as a
real-time feedback to update the latent variables of quantum fake data
$\lvert\phi\rangle$.

The evolution of a quantum system can be expressed in terms of a completely
positive quantum dynamical map $\varepsilon$ as
$\varepsilon(\rho)=\Sigma_{m,n=0}^{d^{2}-1}\chi_{mn}E_{m}\rho
E_{n}^{{\dagger}}$, where $\rho$ is the incident initial state, $\\{E_{m}\\}$
are a set of operator basis elements, $\\{\chi_{mn}\\}$ are the matrix
elements of the superoperator that encodes all the information about the
dynamics. Typically, pauli-matrix rotation quantum processes can be
implemented using wave plates mohseni2006direct; mohseni2007direct. By
applying a general term, an arbitrary combination of unitary matrices turns
the initial state into $\lvert\psi\rangle_{t}=(\alpha\lvert
H\rangle+\beta\lvert V\rangle)/\sqrt{2}$, where the complex probability
amplitudes satisfied the normalization condition as
$|\alpha|^{2}+|\beta|^{2}=1$. Then our self-guided QGAN method can be used to
learn this quantum state with high accuracy as shown in Fig. 2, which
consequently reveals the the corresponding matrix elements of
$\\{\chi_{mn}\\}$.

The single photons are prepared via spontaneous parametric down-conversion
zhong2018photon. In our experimental realization, a 5-mm-long ppKTP crystal is
pumped with a continuous-wave pump laser, which provides type-II collinear
phase matching with pump (p), signal (s), and idler (i) photons at center
wavelengths of $\lambda_{p}\approx 405\,\mathrm{nm}$ and $\lambda_{s,i}\approx
810\,\mathrm{nm}$ at a temperature of $28^{\degree}C$. The pumping of this
nonlinear crystal for SPDC emission is $\lvert H\rangle_{p}\rightarrow(\lvert
H\rangle_{s}\lvert V\rangle_{i}+\lvert V\rangle_{s}\lvert
H\rangle_{i})/\sqrt{2}$. These photons are guided to a polarizing beam
splitter, which maps the orthogonally polarized photon pairs into two distinct
spatial modes chen2020verification. As a direct result, paired single photons
that are separated in distinct spatial modes are routed to the input ports of
a beam splitter. Thereinto, one of the paired photons pass through a series of
wave plates, which changes the single photons’ state to
$\lvert\psi\rangle_{t}$. Its nonclassical beating can be observed by scanning
the time of arrival of one of the photons incident on this beam splitter,
which constitutes a HOM interferometer. The corresponding interference fringes
can be observed in the twofold coincidences between the two output ports of
the beam splitter. Namely, by detecting the HOM interference, it is possible
to perform a similarity measurement on the incident paired photons without any
requirements of any prior knowledge. In each iteration, we route two
independent single photons in quantum states $\lvert\psi\rangle_{t}$ and
$\lvert\phi_{\pm}\rangle$ to a HOM interferometer. The detection in
coincidence at two output ports reveals the overlap between them, i.e.,
$V_{\pm}\propto|\langle\phi_{\pm}|\psi\rangle_{t}|^{2}$, which can be used to
calculated the current gradient by following Eq. (1). These measurement
results would be sent back to the generator, and form the basis for updating
the latent variables of generated state $\lvert\phi\rangle_{t}$. Since the
maximum overlap means that the coincidence detection reaches its minimum value
(the well-known HOM dip), the self-guided QGAN reaches the Nash equilibrium.
According to the analysis of HOM interference, this indicates that two
incident quantum states are indistinguishable in all degrees of freedom, i.e.,
$\lvert\phi\rangle=\lvert\psi\rangle_{t}$ frederic2017high.

Figure 3: Tracking of the self-guided QGAN. (a-c) Experimental results for
three quantum true states $\lvert\psi\rangle_{t1,t2,t3}$ that are prepared by
rotating a HWP. (a) The snapshots of the system at the particular steps at
iteration 1, 2, 4, 8, 10, 16 in the Bloch sphere representation, in which the
quantum true data and fake data is represented by the green and red Bloch
vectors respectively. (b) The tracking of fidelity during the quantum
adversarial learning process. (c) The density matrices of quantum true states
$\lvert\psi\rangle_{t}$ and experimentally estimated states
$\lvert\phi\rangle_{f}$, where an average fidelity of $99.8\%$ is achieved.
(d-f) Experimental results for three quantum true states
$\lvert\psi\rangle_{t4,t5,t6}$ that are prepared by rotating a HWP and a QWP,
where each panel is the counterpart of (a) to (c) respectively. The
experimentally measured average fidelity reaches $99.3\%$.

Figure 3(a-c) shows the experimental results for learning three random quantum
true states $\lvert\psi\rangle_{t1}$=$\lvert H\rangle$,
$\lvert\psi\rangle_{t2}$=0.90$\lvert H\rangle$+0.44$\lvert V\rangle$,
$\lvert\psi\rangle_{t3}$ = 0.69$\lvert H\rangle$ \+ 0.72$\lvert V\rangle$,
which are prepared by rotating a HWP that is relevant to a single parameter to
be estimated. Additionally, a series of HWP and QWP are used to prepare three
random quantum true states $\lvert\psi\rangle_{t4}$ = 0.94$\lvert H\rangle$ \+
0.34$\lvert V\rangle$, $\lvert\psi\rangle_{t5}$ = 0.75$\lvert H\rangle$ \+
(0.07+0.65i)$\lvert V\rangle$, $\lvert\psi\rangle_{t6}$ = 0.82$\lvert
H\rangle$ \+ (0.57+0.11i)$\lvert V\rangle$, which are relevant to two
independent parameters to be estimated, and the experimental results are
depicted in Fig. 3(d-f). The initial quantum fake state is set as
$\lvert\phi\rangle_{i}=\lvert V\rangle$ in both scenarios. During the self-
guided QGAN method, the trajectories of fidelity are recorded [see Fig.
3(b,e)] instead of characterizing the exact experimental $\lvert\phi\rangle$.
Thereinto, the solid lines and dashed lines represent the average fidelities
of 5 experimental trials and 100 theoretical simulations, and the shaded bands
describe the standard deviation of errors. These final experimental results
are consistent with the theoretical predictions, where the slight deviations
can be mainly attributed to the precision of detector and the randomness of
stochastic gradient descent algorithm. The snapshots of the quantum states and
measurement axis at the particular steps derived from the target states
$\lvert\psi\rangle_{t1}$ and $\lvert\psi\rangle_{t4}$ are plotted on the Bloch
sphere as shown in Fig. 3(a,d). While the generator learns from the
measurement outcomes and follows stochastic gradient descent algorithm, the
generated quantum fake data (shown as the red Bloch vectors) gradually
converges to the quantum true data (shown as the green Bloch vectors). As a
direct result, the discriminator that use HOM interference ultimately fails to
discriminate $\lvert\phi\rangle_{f}$ and $\lvert\psi\rangle_{t}$, and
correspondingly the generator achieves its goal of replicating the statistics
of the quantum true data. In order to evaluate the performance of self-guided
QGAN, we analyze the state fidelity to quantify the indistinguishability
between the quantum true data and the generated data as
$F(\phi_{f},\psi_{t})=tr\sqrt{\sqrt{\psi_{t}}\phi_{f}\sqrt{\psi_{t}}}$. As
shown in Fig. 3(c,f), the average fidelity of $99.6\%$ is achieved after 20
iterations in our proof-of-principle experiment, which can be further enhanced
by increasing the number of iterations and improving the experimental
implementation such as higher signal-to-noise ration or higher detection
efficiency. We note that our approach can reach its equilibrium without any
requirement of prior knowledge about the quantum true data or the HOM
interference measurement performed by the discriminator, and thus promises a
double-blind quantum machine learning process just as its classical
counterparts.

The performance of self-guided QGAN in the presence of significant
environmental noise is also investigated as shown in Fig. 4. We choose
$\lvert\psi\rangle_{t1}$ and $\lvert\psi\rangle_{t4}$ as the quantum true
states and introduce the significant statistical noise from a source of light
emitting diode. These results show excellent performance in the presence and
absence of noise, which indicates the great robustness of our approach against
deleterious noise.

Figure 4: Performance of self-guided QGAN in the presence of environmental
noise for quantum true states $\lvert\psi\rangle_{t1}$ and
$\lvert\psi\rangle_{t4}$. The dashed lines and solid lines represent the
average fidelities of 5 experimental trials with- and without-noise
respectively, and the shaded bands show the standard deviation of errors.

##  IV Self-guided QGAN for multiple phase estimation

Quantum metrology exploits quantum mechanics to enable higher precision than
its classical counterpart. This has been widely explored with the estimation
of optical phase shifts by means of interferometry providing the dominant
paradigm chen2019hong; lyons2018attosecond. One of the most important
metrology problems to the wider research community is the microscopy and
imaging ndagano2022quantum. Phase imaging is a cornerstone of optical
microscopy, typically realized using the related techniques of phase contrast
and differential interference contrast imaging. Our approach maps phase
imaging onto the problem of multiple phase estimation simultaneously
pezze2017optimal; hong2021quantum; humphreys2013quantum; gorecki2022multiple.
We provide a strategy for the estimation of multiple phases using self-guided
QGAN method, in which the multiparameter nature of the problem leads to an
intrinsic benefit when exploiting quantum resources.

As shown in Fig. 2, since the difference frequency of down-converted photons
typically exceeds that of the pump laser, high-dimensional frequency
entanglement arises quite naturally as a consequence of energy conservation,
and can be written in the form of

$\lvert\Psi\rangle_{\omega}=\sum_{k}^{n}A_{k}(\lvert\omega_{s}^{k}\omega_{i}^{k}\rangle+\lvert\omega_{i}^{k}\omega_{s}^{k}\rangle),$
(4)

where $A_{k}$ represents the probability amplitude of the $k-th$ frequency-bin
entanglement, $n$ donates the dimensions of prepared entanglement,
$\omega_{s,i,p}$ represent the center frequencies of signal, idler and pump
photons respectively, and they satisfy the energy conservation as
$\omega_{s}^{k}+\omega_{i}^{k}=\omega_{p}$. By encoding the quantum true data
that is described by multiple unknown phases
$\\{\psi_{1},\psi_{2},...,\psi_{n}\\}$, a commercial wave shaper can be
readily used to modulate the relative phase shifts on the signal photon
independently. Meanwhile, the generator also uses a wave shaper to encode the
imitated quantum data $\\{\phi_{1},\phi_{2},...,\phi_{n}\\}$ on the paired
idler photons. Accordingly, the initial entangled state is transformed into

$\lvert\Psi\rangle_{\omega}\rightarrow\sum_{k}^{n}A_{k}(\lvert\omega_{s}^{k}\omega_{i}^{k}\rangle+e^{i(\psi_{k}-\phi_{k})}\lvert\omega_{i}^{k}\omega_{s}^{k}\rangle).$
(5)

Then the discriminator uses HOM interference as a projective measurement to
distinguish the quantum true data and the generated data. Specifically, in
each iteration, paired photons are routed to a balanced HOM interferometer,
and the coincidence counts in two opposite spatial modes are identified. The
manifestation of interference fringes can be approximated by the sum of
coincidence probabilities with difference frequency detunings as
chen2021temporal

$P(\tau)=\sum_{k}^{n}\frac{A_{k}}{2}[1-cos(\psi_{k}-\phi_{k})exp(-\sigma^{2}\tau^{2})],$
(6)

where $\sigma$ is the RMS (root mean square) bandwidth of SPDC photons. For
visibility and efficiency, we can set $\tau=0$ such that the $P(\tau)$ is only
determined by the term of $\phi_{k}-\varphi_{k}$. Analogously, the measured
interference probability is used to calculate the current gradient by
following (1), which would be sent back to the generator for updating the
latent variables and forming the basis for next iteration. Since the maximum
overlap means that the coincidence detection reaches its maximum value (the
well-known HOM peak), the self-guided QGAN reaches the Nash equilibrium,
namely two incident quantum states are indistinguishable in all degrees of
freedom, i.e., $\psi_{k}=\phi_{k}$ for all $k=1,2,...,n$ simultaneously.

To evaluate the performance of our self-guided QGAN for multiphase estimation
simultaneously, Fig. 5 demonstrates the theoretical simulation results,
wherein the estimation accuracy can reach $0.98$ for the estimation of $100$
unknown phases simultaneously. We note again that the gradient estimation step
and convergence can be tuned by the corresponding parameters in the stochastic
gradient descent algorithm.

Figure 5: Theoretical simulation of self-guided QGAN for simultaneous
estimation of 10, 20, 50, 70 and 100 phases.

##  V Discussion

We present the concept and experiment of self-guided QGAN for simultaneous
multiparameter estimation with the assistance of adaptive feedback that is
based on stochastic gradient descent algorithm. Thereinto, the quantum true
data and the generated fake data are all encoded using quantum superposition
states, and the discriminator performs the HOM interference as the projective
measurement. We also explore the robustness of self-guided QGAN against
deleterious experimental and environmental noise, and show excellent stability
as a direct result of quantum interference measurement.

While the self-guided QGAN approach has be used for adaptive tomography of
process dynamics and simultaneous estimation of multiple phases as
demonstrated in this work, it can be readily extended to analogous
applications like quantum image generation huang2021experimental and quantum
state tomography rambach2021robust. Additionally, the self-guided QGAN
algorithm can be straightforwardly extended to a quantum system with higher
dimensions without any requirements of modification of the experimental setup,
which consequently paves the way to tackle more complicated information
processing tasks. In our architecture, the bosonic mode actually provides a
quantum system with infinite dimensions, the whole approach can be extended to
other platform like superconducting circuit hu2019quantum. Another possible
extension of our current experiment is to explore a more complicated
architecture with multiple photonic modes, wherein the two-photon HOM
interferometric configuration should be replaced with multiport quantum
interference zhong2018photon. Our work may inspire more theoretical and
experimental researches into the advanced quantum machine learning algorithms
that provides great advantages over the classical methods.

## Acknowledgements

This work is supported by the National Natural Science Foundation of China
(NSFC) (12034016, 12004318, 61975169), the Fundamental Research Funds for the
Central Universities at Xiamen University (20720190057, 20720210096), the
Natural Science Foundation of Fujian Province of China (2020J05004), the
Natural Science Foundation of Fujian Province of China for Distinguished Young
Scientists (2015J06002), and the program for New Century Excellent Talents in
University of China (NCET-13-0495).

## References

  * Salimans  _et al._ (2016) T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, and X. Chen, in  _Advances in Neural Information Processing Systems_, Vol. 29, edited by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Inc., 2016).
  * Wang  _et al._ (2018) J. Wang, W. Wang, and W. Gao, IEEE Transactions on Multimedia 20, 2578 (2018).
  * Ledig  _et al._ (2017) C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, _et al._ , in  _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2017) pp. 4681–4690.
  * Heusel  _et al._ (2017) M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, in  _Advances in Neural Information Processing Systems_, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., 2017).
  * Sagiroglu and Sinanc (2013) S. Sagiroglu and D. Sinanc, in  _2013 International Conference on Collaboration Technologies and Systems (CTS)_ (2013) pp. 42–47.
  * Marx (2013) V. Marx, Nature 498, 255 (2013).
  * Biamonte  _et al._ (2017) J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature 549, 195 (2017).
  * Lloyd and Weedbrook (2018) S. Lloyd and C. Weedbrook, Phys. Rev. Lett. 121, 040502 (2018).
  * Dallaire-Demers and Killoran (2018) P.-L. Dallaire-Demers and N. Killoran, Phys. Rev. A 98, 012324 (2018).
  * Zoufal  _et al._ (2019) C. Zoufal, A. Lucchi, and S. Woerner, npj Quantum Information 5, 1 (2019).
  * Hu  _et al._ (2019) L. Hu, S.-H. Wu, W. Cai, Y. Ma, X. Mu, Y. Xu, H. Wang, Y. Song, D.-L. Deng, C.-L. Zou, _et al._ , Science advances 5, eaav2761 (2019), https://www.science.org/doi/pdf/10.1126/sciadv.aav2761 .
  * He  _et al._ (2016) K. He, X. Zhang, S. Ren, and J. Sun, in  _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2016) pp. 770–778.
  * Bishop and Nasrabadi (2006) C. M. Bishop and N. M. Nasrabadi,  _Pattern recognition and machine learning_, Vol. 4 (Springer, 2006).
  * Duchi  _et al._ (2011) J. Duchi, E. Hazan, and Y. Singer, Journal of machine learning research 12 (2011).
  * Goodfellow  _et al._ (2014) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, in  _Advances in Neural Information Processing Systems_, Vol. 27, edited by Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger (Curran Associates, Inc., 2014).
  * Mohseni and Lidar (2006) M. Mohseni and D. A. Lidar, Phys. Rev. Lett. 97, 170501 (2006).
  * Mohseni and Lidar (2007) M. Mohseni and D. A. Lidar, Phys. Rev. A 75, 062331 (2007).
  * Graham  _et al._ (2013) T. M. Graham, J. T. Barreiro, M. Mohseni, and P. G. Kwiat, Phys. Rev. Lett. 110, 060404 (2013).
  * Wang  _et al._ (2007) Z.-W. Wang, Y.-S. Zhang, Y.-F. Huang, X.-F. Ren, and G.-C. Guo, Phys. Rev. A 75, 044304 (2007).
  * Kim  _et al._ (2020) Y. Kim, Y. S. Teo, D. Ahn, D.-G. Im, Y.-W. Cho, G. Leuchs, L. L. Sánchez-Soto, H. Jeong, and Y.-H. Kim, Phys. Rev. Lett. 124, 210401 (2020).
  * Luchnikov  _et al._ (2020) I. A. Luchnikov, S. V. Vintskevich, D. A. Grigoriev, and S. N. Filippov, Phys. Rev. Lett. 124, 140502 (2020).
  * Shabani  _et al._ (2011) A. Shabani, R. L. Kosut, M. Mohseni, H. Rabitz, M. A. Broome, M. P. Almeida, A. Fedrizzi, and A. G. White, Phys. Rev. Lett. 106, 100401 (2011).
  * Hong  _et al._ (1987) C. K. Hong, Z. Y. Ou, and L. Mandel, Phys. Rev. Lett. 59, 2044 (1987).
  * Nagali  _et al._ (2009) E. Nagali, L. Sansoni, F. Sciarrino, F. De Martini, L. Marrucci, B. Piccirillo, E. Karimi, and E. Santamato, Nature Photonics 3, 720 (2009).
  * Górecki and Demkowicz-Dobrzański (2022) W. Górecki and R. Demkowicz-Dobrzański, Phys. Rev. Lett. 128, 040504 (2022).
  * Liu  _et al._ (2021) L.-Z. Liu, Y.-Z. Zhang, Z.-D. Li, R. Zhang, X.-F. Yin, Y.-Y. Fei, L. Li, N.-L. Liu, F. Xu, Y.-A. Chen, _et al._ , Nature Photonics 15, 137 (2021).
  * Humphreys  _et al._ (2013) P. C. Humphreys, M. Barbieri, A. Datta, and I. A. Walmsley, Phys. Rev. Lett. 111, 070403 (2013).
  * Hong  _et al._ (2021) S. Hong, Y.-S. Kim, Y.-W. Cho, S.-W. Lee, H. Jung, S. Moon, S.-W. Han, H.-T. Lim, _et al._ , Nature communications 12, 1 (2021).
  * Pezzè  _et al._ (2017) L. Pezzè, M. A. Ciampini, N. Spagnolo, P. C. Humphreys, A. Datta, I. A. Walmsley, M. Barbieri, F. Sciarrino, and A. Smerzi, Phys. Rev. Lett. 119, 130504 (2017).
  * Chapman  _et al._ (2016) R. J. Chapman, C. Ferrie, and A. Peruzzo, Phys. Rev. Lett. 117, 040402 (2016).
  * Rambach  _et al._ (2021) M. Rambach, M. Qaryan, M. Kewming, C. Ferrie, A. G. White, and J. Romero, Phys. Rev. Lett. 126, 100402 (2021).
  * Ferrie (2014) C. Ferrie, Phys. Rev. Lett. 113, 190404 (2014).
  * Hou  _et al._ (2020) Z. Hou, J.-F. Tang, C. Ferrie, G.-Y. Xiang, C.-F. Li, and G.-C. Guo, Phys. Rev. A 101, 022317 (2020).
  * Guo  _et al._ (2017) X. Guo, Y. Mei, and S. Du, Optica 4, 388 (2017).
  * Bulla  _et al._ (2022) L. Bulla, M. Pivoluska, K. Hjorth, O. Kohout, J. Lang, S. Ecker, S. P. Neumann, J. Bittermann, R. Kindler, M. Huber, _et al._ , arXiv preprint arXiv:2204.07536 (2022), 10.48550/ARXIV.2204.07536.
  * Vedovato  _et al._ (2018) F. Vedovato, C. Agnesi, M. Tomasin, M. Avesani, J.-A. Larsson, G. Vallone, and P. Villoresi, Phys. Rev. Lett. 121, 190401 (2018).
  * Marcikic  _et al._ (2004) I. Marcikic, H. de Riedmatten, W. Tittel, H. Zbinden, M. Legré, and N. Gisin, Phys. Rev. Lett. 93, 180502 (2004).
  * Ndagano  _et al._ (2022) B. Ndagano, H. Defienne, D. Branford, Y. D. Shah, A. Lyons, N. Westerberg, E. M. Gauger, and D. Faccio, Nature Photonics , 1 (2022).
  * Xia and Kais (2018) R. Xia and S. Kais, Nature communications 9, 4195 (2018).
  * Schuld and Killoran (2019) M. Schuld and N. Killoran, Phys. Rev. Lett. 122, 040504 (2019).
  * Cai  _et al._ (2015) X.-D. Cai, D. Wu, Z.-E. Su, M.-C. Chen, X.-L. Wang, L. Li, N.-L. Liu, C.-Y. Lu, and J.-W. Pan, Physical review letters 114, 110504 (2015).
  * Chen  _et al._ (2018) Y. Chen, S. Ecker, S. Wengerowsky, L. Bulla, S. K. Joshi, F. Steinlechner, and R. Ursin, Phys. Rev. Lett. 121, 200502 (2018).
  * Zhong  _et al._ (2018) H.-S. Zhong, Y. Li, W. Li, L.-C. Peng, Z.-E. Su, Y. Hu, Y.-M. He, X. Ding, W. Zhang, H. Li, L. Zhang, Z. Wang, L. You, X.-L. Wang, X. Jiang, L. Li, Y.-A. Chen, N.-L. Liu, C.-Y. Lu, and J.-W. Pan, Phys. Rev. Lett. 121, 250505 (2018).
  * Chen  _et al._ (2020) Y. Chen, S. Ecker, J. Bavaresco, T. Scheidl, L. Chen, F. Steinlechner, M. Huber, and R. Ursin, Phys. Rev. A 101, 032302 (2020).
  * Bouchard  _et al._ (2017) F. Bouchard, R. Fickler, R. W. Boyd, and E. Karimi, Science Advances 3, e1601915 (2017), https://www.science.org/doi/pdf/10.1126/sciadv.1601915 .
  * Chen  _et al._ (2019) Y. Chen, M. Fink, F. Steinlechner, J. P. Torres, and R. Ursin, npj Quantum Information 5, 1 (2019).
  * Lyons  _et al._ (2018) A. Lyons, G. C. Knee, E. Bolduc, T. Roger, J. Leach, E. M. Gauger, and D. Faccio, Science advances 4, eaap9416 (2018).
  * Chen  _et al._ (2021) Y. Chen, S. Ecker, L. Chen, F. Steinlechner, M. Huber, and R. Ursin, npj Quantum Information 7, 1 (2021).
  * Huang  _et al._ (2021) H.-L. Huang, Y. Du, M. Gong, Y. Zhao, Y. Wu, C. Wang, S. Li, F. Liang, J. Lin, Y. Xu, R. Yang, T. Liu, M.-H. Hsieh, H. Deng, H. Rong, C.-Z. Peng, C.-Y. Lu, Y.-A. Chen, D. Tao, X. Zhu, and J.-W. Pan, Phys. Rev. Applied 16, 024051 (2021).