# Color Recommendation for Vector Graphic Documents based on Multi-Palette
Representation

Qianru Qiu
CyberAgent
Shibuya, Tokyo, Japan
<EMAIL_ADDRESS>Xueting Wang
CyberAgent
Shibuya, Tokyo, Japan
<EMAIL_ADDRESS>Mayu Otani
CyberAgent
Shibuya, Tokyo, Japan
<EMAIL_ADDRESS>Yuki Iwazaki
CyberAgent
Shibuya, Tokyo, Japan
<EMAIL_ADDRESS>
###### Abstract

Vector graphic documents present multiple visual elements, such as images,
shapes, and texts. Choosing appropriate colors for multiple visual elements is
a difficult but crucial task for both amateurs and professional designers.
Instead of creating a single color palette for all elements, we extract
multiple color palettes from each visual element in a graphic document, and
then combine them into a color sequence. We propose a masked color model for
color sequence completion and recommend the specified colors based on color
context in multi-palette with high probability. We train the model and build a
color recommendation system on a large-scale dataset of vector graphic
documents. The proposed color recommendation method outperformed other state-
of-the-art methods by both quantitative and qualitative evaluations on color
prediction and our color recommendation system received positive feedback from
professional designers in an interview study.

Figure 1: Color recommendation for vector graphic documents. On the left are
the design sample and its visual elements (e.g. Image, SVG and Text) with
extracted palettes. On the right are the recommended top3 results recoloring
one SVG color. The elements in the input sample are from the Crello dataset.

##  1 Introduction

In graphic design, there are many creative applications providing thousands of
templates. These design platforms are suitable for creative designers and
amateurs such as marketing professionals, bloggers, social media managers,
etc. In design workflows, users choose a template and replace the elements
using their own assets. The pre-designed templates have coordinated colors in
each visual element. When some visual elements are replaced, the color harmony
may be destroyed. Selecting appropriate colors is not easy for amateurs, even
designers usually struggle with getting suitable color palettes for vector
graphic documents.

A color palette refers to a limited number of colors expressed in refined
forms. It is widely used in graphic design due to its simplicity,
intuitiveness, generality, and easy computation [12]. The prior researches of
color palette representation proposed to train a regression model [16, 13].
These regression methods extract hundreds of color features manually and learn
the weights of each feature. The feature extraction is complex including
palette colors, mean, standard deviation, median, max, min, and max minus min
across a single channel in each color space, i.e., RGB, CIELAB, and HSV. The
difficulty of the hand-crafted features is that they do not comprehensively
encode the semantics of colors, and some features might not have significant
effects in terms of the downstream tasks. Learning high-quality representation
of color remains an open problem. In this work, we simplify the input without
hand-crafted features and propose a data-driven deep learning model for color
representation.

In recent years, some researchers have explored deep learning techniques on
color palette generation and color recommendation. The previous researches
focus on generating a color palette for a single visual target, such as image
colorization [5], shapes colorization in statistical graphics [14], shapes and
texts colorization in infographics [22]. However, a vector graphic document is
much more complex with multiple visual elements, including images, shapes, and
texts simultaneously. Each visual element has its own palette. It is
challenging for existing color recommendation tools to recommend colors for
multi-palette design. In this study, we use a color sequence combining
multiple palettes of different visual elements and train a masked color model
to learn multi-palette representation by color sequence completion.

In summary, our main contributions include:

  * •

A novel masked color model to represent multiple palettes in vector graphic
documents.

  * •

An interactive system for color recommendation which recolors graphic
documents with the recommended colors.

  * •

A series of experimental evaluations covering quantitative experiments and
perceptual studies for the recommendation system and the recommended results
that validates the effectiveness of the proposed methods.

##  2 Related works

###  2.1 Color recommendation

There are mainly two scenarios of color recommendation. The first one is to
suggest a color palette for specified themes or semantic requirements. The
second one is to expand a color palette based on the given colors. For the
first scenario, there are some websites, such as Adobe Color [1], and
COLOURLovers [2], providing color palette templates classified with various
theme names or semantic tags, such as ’natural’, ’environmental’. These
palette templates can be used as references to suit semantic requirements.
Some researches suggest color palette templates based on semantic tags and
fixed harmonic color selection models to generate text colors based on image
colors in magazine cover design [8, 21]. For the second scenario, an early
effort by O’Donovan _et al_. [16] proposed a linear regression method and
suggested the fifth color for the given four colors. Kita _et al_. [13] used
the same regression method to expand a color palette composed of N colors to
$N+\alpha$ retraining the original color harmony. These regression models
depended on the hand-crafted feature extraction methods. In this work, we
suggest compatible colors for the given colors in multi-palette without color
feature extraction methods by a deep learning model.

Recently, some researchers have explored deep learning algorithms on color
palette recommendation. Yuan _et al_. [22] employed a Variational AutoEncoder
with Arbitrary Conditioning (VAEAC) model to generate a color palette for
infographic elements. Kim _et al_. [11] trained the color embedding model to
predict and recommend other colors that are likely to gather together in the
same palette for Mandala Coloring. The color model was in a similar way as
fastText [6], an extension of Word2Vec [15] model. They signified a color as a
word, and the palette as a sentence. The model was trained providing a
continuous vector representation of colors. The elements in these design
objects include shapes, or texts, and each element is limited to a single
color. In graphic documents, the elements also include photos and
illustrations. The color design in vector graphic documents is more complex as
a multi-palette design. We extend the idea of applying word embedding to color
representation similar to the previous Word2Vec-based work [11]. However, the
Word2Vec models aren’t able to account for the relationships between different
palettes in the same graphic document. We explore multi-palette representation
by contextual embedding based on BERT architecture [9].

###  2.2 Palette-based image recoloring

There are some image recoloring approaches based on semantic segmentation from
image by deep neural networks [3, 10]. We focus on palette-based models that a
color palette is useful and easy to express the color concept of an image.
Color palette captures the main colors in an image and is used to adjust the
color composition of an image towards the desired color palette. Most
approaches for recoloring images involve two steps, as extracting a palette
from the image and mapping every pixel in the image to the target palette.
Many approaches [7, 23, 4] employ clustering methods to extract palette
colors. Several other works [17, 18] use a geometric method to extract
palettes that construct a convex hull in RGB color space. The convex-hull-
based palettes may miss important colors that lie within the convex hull. We
use k-means clustering method to recolor image elements in our system.

##  3 Multi-palette representation

###  3.1 Datasets

We generate a multi-palette dataset from a large-scale dataset Crello [19].
The Crello dataset contains design templates for various display formats, such
as social media posts, banner ads, blog headers, or printed posters. It offers
complete document structure and element attributes including element-specific
configuration, such as type of the element, position, size, opacity, color, or
a raster image. The element types mainly include imageElement, maskElement,
coloredBackground, svgElement, and textElement. We classify the elements into
three groups, as image element group including imageElement and maskElement,
scalable vector graphic (SVG) element group including coloredBackground and
svgElement, and text element group including textElement. The color data of
each element in the Crello dataset only has 1 color that is relevant for solid
background and text placeholder. We generate a multi-palette dataset as Image-
SVG-Text palettes that each element group has its own palette as shown in
Figure 2. For image and SVG elements, we merge the elements of the same group
into a single image and then extract the color palette using k-means
clustering method. For text elements, we collect text colors and cluster them
into a palette. Each palette is up to 5 colors in this work. We get 18,768 /
2,315 / 2,278 valid data as train, validation, and test datasets. All design
templates in the figures of this paper are from the Crello test dataset.

Figure 2: Multi-palette extraction from a design template as Image-SVG-Text
palettes. Merge the elements in the same element group into a single image and
then extract the color palette.

###  3.2 Representation learning with masked color model

Figure 3: Masked color model for image-SVG-text color sequence.

We train the color embedding model in a similar way as the word embedding
model. In natural language processing, the word embedding model is used to
learn distributed representation, where the input is a text corpus and the
output is a set of feature vectors that represent words. Similarly, in the
color embedding model, a color signifies a word, a palette signifies a
sentence, and multiple palettes in the same design signify a paragraph.

For the input color corpus, we adopt CIELAB color space, which is more
perceptually uniform than other color spaces [12]. The most general-used color
space is the 24-bit RGB model. We convert RGB color data to CIELAB with a
range of [0, 255], and assign each color to one of the bins in a $b\times
b\times b$ histogram (we use $b=16$ in this work). For example, the color
white(255, 255, 255) in RGB color space is labeled as the code
${}^{\prime}15\\_8\\_8^{\prime}$ in CIELAB color space with 16 bins. There are
796 color codes in the vocabulary of the train dataset. The color codes are
converted into vectors and embedded in the space in the learning progress.

We obtain color embeddings by a masked color model based on pre-training BERT
architecture [9]. The masked color model in Figure 3 is trained in the similar
way as masked language model (Masked LM) in BERT. The model receives a fixed
length of each palette as input. For a palette that is shorter than this fixed
length, we will have to add the token [PAD] to the palette to make up the
length. Another artificial token [SEP], is added to the end of a palette. For
a given token, its input representation is constructed by summing the
corresponding token, segment, and position embeddings. Here,
$\\{C_{1_{1}},\ldots C_{1_{5}}\\}$ is for image color palette,
$\\{C_{2_{1}},\ldots C_{2_{4}}\\}$ is for SVG color palette,
$\\{C_{3_{1}},C_{3_{2}}\\}$ is for text color palette. The palettes of image,
SVG, and text, are respectively labeled with the segment number 1, 2, and 3.
The segment embeddings are basically the palette number that is encoded into a
vector. The trained model knows whether a particular color token belongs to a
specific palette. The multi-palette representation can be achieved by segment
embedings.

The masked color model randomly masks some percentage of the tokens from the
input, and then predict the masked tokens based on its context. In our
experiments, we mask $10\%$ of the tokens in each sequence at random and
replace the chosen token with the [MASK] token $80\%$ of the time. Then, we
use the standard cross entropy loss to optimize the pre-training task. The
final embeddings from the Transformer self-attention mechanism can be used in
downstream tasks, such as aesthetic rating prediction. In current work, we
recommend colors in multi-palette by predicting masked colors with high
probability.

###  3.3 Color recommendation system

In the existing creative applications for graphic documents, users are allowed
to choose and edit a design template. However, when same visual elements are
replaced, users may struggle with coordinating the colors in the design. To
reduce user’s work, we propose a system to recommend the specified colors and
recolor the elements with the recommended colors automatically.

We create color recommendation engine with masked color model and develop an
interactive user interface that enables users to obtain the coordinated colors
for design. The color recommendation system supports basic selecting,
interactive recommendation, and previewing functions shown in Figure 4. The
design template is converted to a JSON file as the system input that contains
complete element-specific configuration. The system parses a JSON object and
reconstructs the design with separated visual elements. The system allows
users to change the image elements and extracts the color palette of each
element group. Users can select the colors for recoloring and then check the
design results with the recommended colors. For SVG recoloring, the original
color is changed to the recommended color by a simple interpolation method.
For image recoloring, we use a palette-based photo recoloring method by
k-means clustering [7] in this system.

Figure 4: Interactive interface of color recommendation system for vector
graphic documents contains six main operations: \raisebox{-0.9pt}{1}⃝ Input a
JSON file. \raisebox{-0.9pt}{2}⃝ Replace image elements. \raisebox{-0.9pt}{3}⃝
Get image-SVG-text palettes and select the colors for recoloring.
\raisebox{-0.9pt}{4}⃝ Get the recommended colors from the color recommendation
engine. \raisebox{-0.9pt}{5}⃝ Choose a recommended color and check the
recolored result. \raisebox{-0.9pt}{6}⃝ Mark the preferred results.

##  4 Experimental validation

To evaluate the performance of our proposed approach, we compare with related
work and a baseline model by both quantitative and qualitative evaluations. We
adapted a Word2Vec-based model which is used in the related work [11]. The
input for this model is the color token without segment embeddings. We also
trained a BERT-based model without segment embeddings as a baseline to show
the effectiveness of the segment embeddings for multi-palette representation.

###  4.1 Quantitative evaluation

We use 2278 color sequences of our test dataset in the quantitative
experiments. We mask a color in color sequence randomly and evaluate the
accuracy of the predicted color. We use top N accuracy that the true color is
equal to any of the N most probable colors predicted by each model. The human
eye sometimes cannot fully perceive subtle differences in color. In addition
to accuracy, we also use visual similarity to measure the recommended colors.
For similarity measurement, we calculate the distance between two colors using
CIEDE2000 rather than Euclidean distance, which has exhibited good performance
in predicting visual similarity between color palettes [20, 12].

Firstly, we train 20 times and get the mean value of accuracy. The results of
our model with and without segment and position embeddings are shown in Table
1. It is found that there is no difference between the models with and without
position embeddings in the current dataset. Thus we pick up the best models
trained by our method without position embeddings for following comparisons.

Embeddings | Accuracy$\uparrow$
---|---
segment | position | @1 | @3 | @5 | @10
w/ | w/ | 0.27 | 0.45 | 0.53 | 0.63
w/ | w/o | 0.27 | 0.44 | 0.52 | 0.62
w/o | w/o | 0.16 | 0.30 | 0.38 | 0.50
Table 1: Quantitative comparison of our model with and without segment and
position embeddings on top N accuracy (N = 1, 3, 5, 10). Here is the mean
value of 20 trained models.

To compare our method with the Word2Vec-based model and the baseline model, we
evaluate the accuracy and similarity of color prediction results by these
three models. The comparison results of accuracy are shown in Figure 5, Table
2, and the comparison results of similarity are shown in Figure 6, Table 3.
Our method with segment embeddings provides significantly better results than
the Word2Vec-based method and the baseline model. The results show that the
segmentation is effective in multi-palette representation learning and brings
better performance to color recommendation. We suggest providing more than two
color candidates in recommendation applications with a high accuracy and users
would like to find a desired color in top N recommended colors.

Figure 5: Quantitative comparison of our models with and without segment embeddings and the Word2Vec-based model on top N accuracy (N = 1, 2, 3, 4, 5). | Accuracy$\uparrow$
---|---
Models | @1 | @2 | @3 | @4 | @5
Word2Vec | 0.03 | 0.05 | 0.08 | 0.10 | 0.11
Ours w/o segment | 0.23 | 0.32 | 0.39 | 0.43 | 0.46
Ours w/ segment | 0.36 | 0.46 | 0.52 | 0.57 | 0.61
Table 2: Quantitative comparison of our models with and without segment embeddings and the Word2Vec-based model on top N accuracy (N = 1, 2, 3, 4, 5). Figure 6: Quantitative comparison of our models with and without segment embeddings and the Word2Vec-based model on top N similarity (N = 1, 2, 3, 4, 5). | Similarity$\downarrow$
---|---
Models | @1 | @2 | @3 | @4 | @5
Word2Vec | 38.4 | 28.3 | 23.9 | 20.3 | 17.8
Ours w/o segment | 30.6 | 18.8 | 14.1 | 11.5 | 9.9
Ours w/ segment | 23.8 | 14.5 | 10.7 | 8.7 | 7.4
Table 3: Quantitative comparison of our models with and without segment
embeddings and the Word2Vec-based model on top N similarity (N = 1, 2, 3, 4,
5).

###  4.2 Qualitative evaluation

Figure 7: Hue distribution of selected colors in the evaluation samples and
all colors in the Crello test dataset. Hue orders are based on the Practical
Color Co-ordinate System, and we denote the neutral color as -1. Figure 8:
Color recommendation results with our proposed BERT-based models with and
without segment embeddings, and the Word2Vec-based model. The three samples
are recolored with one image color, one SVG color, and one text color
respectively.

Considering that color performance depends on human perception, we conducted
qualitative evaluation to verify the performance of the recommended results.
We randomly selected 80 templates from the Crello test dataset and randomly
selected one color for recoloring in multi-palette, that can be in image, or
SVG, or text element. The hue distribution of selected color in the evaluation
samples and all colors in the test dataset are shown in Figure 7. The neutral
colors in some visually imperceptible elements are excluded during random
selection, e.g. a text color with very small font size is ignored in this
experiment. The recolored designs by the top1 recommended colors of each model
are shown in Figure 8.

We pick up one original design (GT), and the top2 recommended results from
three models including our model with segment, the baseline model without
segment, and the Word2Vec-based model. These seven designs are arranged
together in an evaluation question. The participants are asked to select at
most three good designs and three bad designs from seven designs. We recruited
84 participants with 68 non-designers and 16 graphic designers.

For non-designers, the evaluation results of good and bad design selections
are shown in Figure 9 and Figure 10. Here is the mean value of top2
recommendation results by three models. We can find that though the results by
our proposed model with segment embeddings get worse performance than GT, they
have higher preference and lower dislike than the baseline model without
segment and the Word2Vec-based model ($p<0.1$). There is no significant
difference between the baseline model and the Word2Vec-based model.

Figure 9: Evaluation results of good design from non-designers. Figure 10:
Evaluation results of bad design from non-designers.

For designers, the evaluation results of good and bad design selections are
shown in Figure 11 and Figure 12. The results are similar with the results
from non-designers. Moreover, designers evaluate that our model with segment
has significantly better performance than the Word2Vec model (preference:
$p<0.001$, dislike: $p<0.01$). From the evaluation results of non-designers
and designers, the most obvious finding is that our proposed model has better
performance than the Word2Vec-based model. Moreover, the segment embedding is
necessary to multi-paltte representation.

Figure 11: Evaluation results of good design from designers. Figure 12:
Evaluation results of bad design from designers.

###  4.3 Interview study

To access the color recommendation system as in Figure 4, we collected
qualitative feedback from 12 professional designers aged 20-39. We provided a
short tutorial of our system and asked the participants to explore color
recommendation for recoloring one color in visual elements. For the elements
in some templates, there are more than one dominant colors. We also asked
participants to explore the recommendation of more than one color. Here is a
sample of recoloring three colors in SVG elements in Figure 13. We prepared
design templates and some image samples from the Crello test dataset.

Figure 13: Color recommendation results for recoloring three colors in SVG
elements. The number shown on the palette color is the recommended ranking.

Q1 Struggle with choosing colors in creative design works |
---|---
Q2 Color recommendation tool is useful |
Q3 This recommendation system is easy to use |
Q4 Recommended results for one color look good |
Q5 Recommended results for more than one color look good |
Q6 Like to use this recommendation system for work |
|

Table 4: Evaluation results from 12 designers in the interview study.

We asked participants to replace the image elements in design templates with
image samples. The participants could also upload their own images. The
participants filled out a questionnaire after some trials. There are six
questions and each question has five choices, as strongly agree, agree,
neither, disagree, and strongly disagree.

The results are shown in Table 4. We observed the high demand for color
recommendation systems in creative design works as in Q1 and $91.7\%$
participants answered that a color recommendation tool is useful for graphic
design work in Q2. For Q3 and Q6, $58.3\%$ participants answered that our
current system is easy to use, and $75\%$ participants liked to use our
recommendation system for work. For the recommendation results, $66.7\%$
participants can get a satisfied recommended colors by our system in one color
and more than one color recoloring tasks as in Q4 and Q5. Generally, we
received positive feedback for our color recommendation system and the
recommended results from designers.

###  4.4 Limitations

Accuracy decreases when masked color number increases. We generate the color
sequence with a maximum 15 colors and train our model with masking 10$\%$ of
the token in each sequence. That is, only 1 color is masked for prediction in
most sequences in the training process. When the masked color number
increases, the prediction accuracy decreases significantly in Table 5.

Masked colors | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Accuracy@1$\uparrow$ | 0.36 | 0.29 | 0.24 | 0.20 | 0.17
Table 5: Top 1 accuracy for predicting different numbers of masked colors.

Lack of diversity in recommended colors when more than one color is masked.
Though users can freely combine the recommended colors and designers can get a
satisfactory design by our recommendation system in interview study, how to
recommend a complete palette for each element by a learned model remains a
problem. The recommended colors in the same element group are highly similar
as in Figure 13. Furthermore, neutral colors have a semantic value to a color
palette and we do not filter out these high-frequency colors. Neutral colors
have high frequency in the dataset, and it has higher probability to be
recommended than chromatic colors.

##  5 Conclusions

We proposed a masked color model for multi-palette representation to recommend
colors for vector graphic documents and developed an interactive system of
recoloring the specified colors in visual elements. The performance of the
proposed system is experimentally verified through both quantitative and
qualitative evaluations compared to the state-of-the-art method of a Word2Vec-
based model and the baseline model. To our knowledge, our method opens the
door to recommend colors for vector graphic design based on multi-palette of
visual elements. We will explore to improve the performance of a complete
palette recommendation and fine-turn our model for the practical applications
in the future work.

## References

  * [1] Adobe color.  https://color.adobe.com/.
  * [2] Colourlovers.  https://www.colourlovers.com/.
  * [3] Mahmoud Afifi, Brian L Price, Scott Cohen, and Michael S Brown.  Image recoloring based on object color distributions.  In Eurographics (Short Papers), pages 33–36, 2019.
  * [4] Naofumi Akimoto, Huachun Zhu, Yanghua Jin, and Yoshimitsu Aoki.  Fast soft color segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8277–8286, 2020.
  * [5] Hyojin Bahng, Seungjoo Yoo, Wonwoong Cho, David Keetae Park, Ziming Wu, Xiaojuan Ma, and Jaegul Choo.  Coloring with words: Guiding image colorization through text-based palette generation.  In Proceedings of the european conference on computer vision (eccv), pages 431–447, 2018.
  * [6] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  Enriching word vectors with subword information.  Transactions of the association for computational linguistics, 5:135–146, 2017.
  * [7] Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein.  Palette-based photo recoloring.  ACM Trans. Graph., 34(4):139–1, 2015.
  * [8] Ali Jahanian, Jerry Liu, Qian Lin, Daniel Tretter, Eamonn O’Brien-Strain, Seungyon Claire Lee, Nic Lyons, and Jan Allebach.  Recommendation system for automatic design of magazine covers.  In Proceedings of the 2013 international conference on Intelligent user interfaces, pages 95–106, 2013.
  * [9] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [10] Siavash Khodadadeh, Saeid Motiian, Zhe Lin, Ladislau Boloni, and Shabnam Ghadar.  Automatic object recoloring using adversarial learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1488–1496, 2021.
  * [11] Eunseo Kim, Jeongmin Hong, Hyuna Lee, and Minsam Ko.  Colorbo: Envisioned mandala coloringthrough human-ai collaboration.  In 27th International Conference on Intelligent User Interfaces, pages 15–26, 2022.
  * [12] Suzi Kim and Sunghee Choi.  Dynamic closest color warping to sort and compare palettes.  ACM TRANSACTIONS ON GRAPHICS, 40(4), 2021.
  * [13] Naoki Kita and Kazunori Miyata.  Aesthetic rating and color suggestion for color palettes.  In Computer Graphics Forum, volume 35, pages 127–136. Wiley Online Library, 2016.
  * [14] Kecheng Lu, Mi Feng, Xin Chen, Michael Sedlmair, Oliver Deussen, Dani Lischinski, Zhanglin Cheng, and Yunhai Wang.  Palettailor: Discriminable colorization for categorical data.  IEEE Transactions on Visualization and Computer Graphics, 27(2):475–484, 2020.
  * [15] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [16] Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann.  Color compatibility from large datasets.  In ACM SIGGRAPH 2011 papers, pages 1–12. 2011.
  * [17] Jianchao Tan, Jose Echevarria, and Yotam Gingold.  Efficient palette-based decomposition and recoloring of images via rgbxy-space geometry.  ACM Transactions on Graphics (TOG), 37(6):1–10, 2018.
  * [18] Yili Wang, Yifan Liu, and Kun Xu.  An improved geometric approach for palette-based image decomposition and recoloring.  In Computer Graphics Forum, volume 38, pages 11–22. Wiley Online Library, 2019.
  * [19] Kota Yamaguchi.  Canvasvae: Learning to generate vector graphic documents.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5461–5469. IEEE, 2021.
  * [20] Jie Yang, Yun Chen, Stephen Westland, and Kaida Xiao.  Predicting visual similarity between colour palettes.  Color Research & Application, 45(3):401–408, 2020.
  * [21] Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, and Shipeng Li.  Automatic generation of visual-textual presentation layout.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 12(2):1–22, 2016.
  * [22] Linping Yuan, Ziqi Zhou, Jian Zhao, Yiqiu Guo, Fan Du, and Huamin Qu.  Infocolorizer: Interactive recommendation of color palettes for infographics.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [23] Qing Zhang, Chunxia Xiao, Hanqiu Sun, and Feng Tang.  Palette-based image recoloring using color decomposition optimization.  IEEE Transactions on Image Processing, 26(4):1952–1964, 2017.


# Experimenting quantum phenomena on NISQ computers using high level quantum
programming

Duc M. Tran  Nano and Energy Center, VNU University of Science, Vietnam
National University, 120401 Hanoi, Vietnam  Duy V. Nguyen  Phenikaa Institute
for Advanced Study, Phenikaa University, 12116 Hanoi, Vietnam  Faculty of
Computer Science, Phenikaa University, 12116 Hanoi, Vietnam  Le Bin Ho  Center
of Physics, Institute of Applied Mechanics and Informatics, Vietnam Academy of
Science and Technology, Ho Chi Minh City 70000, Vietnam  Research Institute of
Electrical Communication, Tohoku University, Sendai 980-8577, Japan  Hung Q.
Nguyen<EMAIL_ADDRESS>Nano and Energy Center, VNU University of Science,
Vietnam National University, 120401 Hanoi, Vietnam

###### Abstract

We execute the quantum eraser, the Elitzur-Vaidman bomb, and the Hardy’s
paradox experiment using high-level programming language on a generic, gate-
based superconducting quantum processor made publicly available by IBM. The
quantum circuits for these experiments use a mixture of one-qubit and multi-
qubit gates and require high entanglement gate accuracy. The results aligned
with theoretical predictions of quantum mechanics to high confidence on
circuits using up to 3 qubits. The power of quantum computers and high-level
language as a platform for experimenting and studying quantum phenomena is
henceforth demonstrated.

## Introduction

Recent advances in quantum engineering have brought about the realization of
quantum computers that perform key quantum algorithms satisfactorily and
display gigantic potentials [1, 2]. This powerful Turing machine employs
quantum superposition and entanglement to perform calculations. By entangling
multiple qubits together, techniques such as quantum parallelism is achieved
to provide exponential speedup for quantum algorithms over classical ones.
Despite prospective impacts, it is tremendously difficult to build a practical
quantum computer that is fault-tolerant [3]. The main technical obstacle lies
in the fact that a quantum state is fragile to noise. It is hard to engineer a
quantum system that is both well-controlled and long-lived. Termed NISQ for
Noisy Intermediate Scale Quantum [4], these prototypes are not ready for real-
life applications that require complexity and precision lying beyond our
current state of the art.

At the moment, the leading candidate in hardware is perhaps superconducting
circuits [5]. At its heart, the qubit comprises a collective state of
electrons, condensed into a single quantum state, the Cooper pairs. This
artificial quantum object can be freely designed and engineered, conveniently
controlled using microwave pulses, and scalable using established nano-
fabrications techniques. In this system, for example of a charge qubit [6],
the quantum superposition is realized by a microwave pulse that places the
tiny superconducting island at a potential that supports both $N$ and $N+1$
number of charges at the same time. This action is equivalent to a
beamsplitter in a Mach-Zehnder interferometer (MZI) that splits the photon’s
route into two uncertain paths. In the quantum circuit language, this is
performed by a Hadamard gate with matrix form
$H=\frac{1}{\sqrt{2}}\bigl{[}\begin{smallmatrix}1&1\\\
1&-1\end{smallmatrix}\bigr{]}$. In a transmon quantum processor [7], such as
those provided by IBM [8], the error of a Hadamard gate is in the range of
$10^{-3}$ to $10^{-4}$. Designed as a universal computing platform and
programmed using high-level language through cloud access, there is a range of
remarkable works performed on real devices with few qubits. They include the
demonstration of important quantum algorithms [9, 10], simulation of quantum
phenomena [11, 12, 13, 14, 15, 16], or reproducing foundation quantum
experiences [17, 18, 19, 20, 21].

Quantum physics is a generic theory that applies to any physical objects, such
as photons, electrons, or artificial devices engineered in the lab. However,
lying on the surface of a chip, the superconducting qubits have limited
connectivity between neighbors. It is unclear if the entanglement of a
collective quantum state induced by a metallic coupler is the same as that of
genuine nature elements. In this work, we perform three fundamental
experiments - those at the core of quantum physics - by programming the IBM
NISQ computers. Using Python scripts written in Qiskit, we demonstrate that
the quantum eraser, the Elitzur-Vaidman bomb, and the Hardy’s paradox
experiments can be reproduced successfully on the same quantum processor.
Originally proposed and realized in optical apparatuses, we showed that
theoretically, these optical setups are equivalent to our quantum circuits.
The quantum circuits are executed on real quantum computers with reasonable
agreement [22]. The experimental deviations to the original results are
associated with the erroneousness in the performance of the superconducting
devices provided by IBM.

## The quantum eraser

The quantum eraser experiment [23, 24, 25] is a natural extension of the
double-slit experiment that demonstrates the counter-intuitive nature of
quantum physics - the wave-particle duality. In Fig. 1 (a, d), the duality is
eavesdropped on by using the Mach-Zehnder interferometer (MZI) apparatus with
a spontaneous parametric down-conversion (SPDC) process. Upon passing the
first beamsplitter (BS), the photon splits into an upper-path (red) and a
lower-path (blue), which indicates the “which-path” information. These paths
encounter the SPDC that generates an entangled “signal-idler” photons pair for
each path, which are denoted as (s) and (i) in Fig. 1 (a, d), respectively.
The signals arrive at the second BS to interfere and are detected by D1, D2
detectors. The idlers are measured by D3 and D4 detectors. In Fig. 1 (a), we
measure them directly after the SPDC, which reveals the red-path or blue-path
information by the clicking of D3 or D4, respectively. In this case, both D1
and D2 click with equal probability, which implies that there is a loss of
interference in the signal photons. By observing the which-path information
via idler photons, the interference in the signal photons is deleted; in other
words, they behave as “particles.” In Fig. 1 (d), the idlers are interfered
with a BS before the measurement. Hence, the which-path information is
“erased.” Jointly measuring D1,2 and D3,4 yields surprising results: only D1
and D3 click coincidentally but not for D2 and D3. It indicates that the
interference in the signal photons produces a constructive signal at D1 and a
destructive signal at D2. Inversely, only D2 and D4 click coincidentally but
not for D1 and D4 detectors. The interference in the signal photons generates
a constructive signal at D2 and a destructive signal at D1. Thus, by erasing
the which-path information, we observe interference in signal photons; in
other words, they behave as “waves.” The wave-particle duality depends on “our
action” on the entangled counterparts. Termed quantum eraser, the phenomenon
is verified for photons [25, 26, 27], or surface acoustic phonons [28].

Figure 1: In the quantum eraser experiment, we recover the interference
pattern by choosing an appropriate measurement basis. (a, d) The quantum
eraser experiments using a Mach-Zehnder interferometer (MZI) and a spontaneous
parametric down-conversion (SPDC), an alternative setup to the double-slit
experiment: upon entering the first beamsplitter (BS), the original photon is
split into upper-path (red) and lower-path (blue). Each path transforms into a
signal-idler photons pair by passing through the SPDC. The signal photons (s)
interfere at the second BS and are measured by D1(2). The idler photons are
measured by D3(4) without an eraser in (a) and with an eraser in (d). (b, e)
Quantum circuit for the experiment in (a, d), respectively. (c, f) Experiment
results obtained from the IBM Vigo quantum computer shown as blue bars and the
theoretical value showed as red lines. The device error is estimated at 3.7%,
while standard error from the mean of 8192 shots is minimal at 0.5%.

This experiment can be written in the quantum circuit language [29] as showed
in Fig. 1 (b, e). Similar to the role of a BS in an optical setup, the first
Hadamard gate splits the first qubit’s initial state $|q_{0}\rangle=|0\rangle$
into $|0\rangle$ and $|1\rangle$ to represent the which-path information:
$|0\rangle$ $\leftrightarrow$ red-path, and $|1\rangle$ $\leftrightarrow$
blue-path. Then, a CNOT gate entangles $|q_{0}\rangle$ with the target
$|q_{1}\rangle$ qubit, which simulates the function of the SPDC. By resembling
$|q_{0}\rangle$ as the signals, a second Hadamard gate is applied on it to
probe the interference patterns. The outcome of this quantum eraser experiment
relies on qubit $|q_{1}\rangle$, which plays the role of the idler photons. In
Fig. 1 (b), $|q_{1}\rangle$ is measured right after the CNOT gate such that
the which-path information can be verified. The measurement results are shown
in Fig. 1 (c), indicating no interference in both $|q_{0}\rangle$ and
$|q_{1}\rangle$. Similarly, in Fig. 1 (e), $|q_{1}\rangle$ is measured after
the action of the Hadamard gate. This is equivalent to the application of a BS
on idler photons in Fig. 1 (d). The which-path information is erased and
interference patterns are detected in $|q_{0}\rangle$, as can be seen from
Fig. 1 (f). Mathematically, it is simple to verify the equivalence between the
quantum circuit and the optical MZI setup. Using matrix multiplication, it is
straightforward to confirm that the circuit in Fig. 1 (b) changes state
$|00\rangle$ to $\frac{1}{2}(|00\rangle+|01\rangle+|10\rangle-|11\rangle$,
while the circuit in Fig. 1 (e) changes $|00\rangle$ to
$\frac{1}{\sqrt{2}}(|00\rangle+|11\rangle$, in agreement with the optical MZI
setup in Fig. 1 (a, d), respectively. These results are confirmed by
simulation results as solid lines on Fig. 1 (c, f) and agree with the original
proposal in the optical apparatus. We execute these circuits on IBM’s Vigo
quantum computer by programming them using Qiskit [22]. In the non-erased
experiment Fig. 1c, the measurement probabilities for $|00\rangle$,
$|01\rangle$, $|10\rangle$, and $|11\rangle$ are $25.4\pm 0.9\%,25.6\pm
0.9\%,22.9\pm 0.8\%,26.1\pm 1.0\%$, respectively. Compared to theoretical
value of 25% for each state, the experiment results are quite closely matched.
Similarly in the erased experiment Fig. 1f, the corresponding probabilities
are $47.0\pm 1.7\%,2.1\pm 0.1\%,2.0\pm 0.1\%,48.9\pm 1.8\%$, which resemble
the theoretical distribution of 50%, 0%, 0%, 50%. It is possible to estimate
errors from our quantum circuits in Fig. 1b,e due to gate and measurement
errors. Using calibration values provided by IBM [22], the uncertainty in our
circuits is 3.7%. We emphasize that a classical interpretation would predict
no recovery of the interference pattern in the erased case, and an equal
distribution to all states in the non-erased case; this was not found. Our
result, therefore, satisfactorily demonstrate the quantum nature of the IBM
machines.

In the quantum circuit representation, the difference between (b) and (e) is
the Hadamard gate on $|q_{1}\rangle$, which changes the measurement basis: in
(b) the measurement is performed on the $z$-basis, while in (e) it is on the
$x$-basis. The wave-particle duality is thus demonstrated by the choice of
measurement basis. This is a feature of our proposed scheme in contrast with
[29], where they encoded the wave and particle behaviors into one photon
(qubit) while entangling with another ancilla photon.

## The Elitzur-Vaidman bomb

Essex Ourense Burlington London Vigo Valencia x2 $\eta$ 0.417 0.387 0.303
0.306 0.356 0.325 0.309 Absolute error 0.084 0.054 0.031 0.027 0.022 0.008
0.024 Relative error 25.1% 16.2% 9.2% 8.1% 6.7% 2.5% 7.3% Standard deviation
0.034 0.047 0.060 0.029 0.017 0.025 0.028 Relative device error 9.3% 4.9% 6.2%
7.4% 4.1% 4.0% 5.3% Absolute device error 0.039 0.019 0.019 0.023 0.015 0.013
0.016

Table 1: Performance of IBM’s quantum devices in our Elitzur-Vaidman bomb
experiments. The averaged efficiency $\eta$ for all machines are listed in the
first row, differ from the theoretical efficiency $\eta=1/3$. The absolute
error is calculated as $|\eta-1/3|$, relative error is
$\frac{|1/3-\eta|}{1/3}$, and standard deviation is
$\sqrt{\sum_{i=1}^{n}\frac{(\eta_{i}-\eta)^{2}}{n}}$, with $n$ is the total
number of runs and $\eta_{i}$ is the efficiency of the $i^{\text{th}}$ run.
Device error is calculated using the fidelity of a circuit, which is estimated
as the product of fidelities of the circuit’s component gates. This
information from IBM can be found in our Github [22] for the time that we ran
these experiments.

The second experiment we address here is the interaction-free measurement,
also known as the Elitzur-Vaidman bomb [30, 31]. In an optical experiment as
shown in Fig. 2 (a), a classical object, such as a light-sensitive bomb, is
placed on one path of the MZI. Due to the non-locality of the photon’s
wavefunction, there is a probability that the bomb is detected without
actually being hit by the photon, thus the name interaction-free measurement.
This experiment has been verified for photon [33], electrons [34], atoms [35],
as well as proposed for solid-state systems [36, 37].

Putting philosophical discussion aside, this experiment can be described as
follows. In a standard MZI without the bomb, the original photon splits into
an upper path and a lower path with 50% chance for each case after the first
BS. It then interferes with itself at the second BS, and always lights up the
constructive detector D2. With the bomb however, if the photon takes the upper
path, it will be absorbed by the bomb, and thus produces no clicks in the
detectors. If the lower path is taken, the photon will split again with an
even chance for each possibility at the second BS, resulting in clicks at D1
and D2 with 25% detection each. While the click on D2 is indistinguishable
from the no-bomb case, a click on detector D1 tells us that there is an active
bomb residing on the upper path inside the MZI. In the latter case, we detect
the existence of the bomb without touching it, hence the name “interaction-
free” measurement.

Figure 2: The Elitzur-Vaidman bomb experiment. (a) The original optical setup
[30] with a light-sensitive bomb inside a MZI. There is a non-zero probability
that a single measurement detects the bomb without an interaction, i.e., the
photon takes the lower path and hits detector D1. More detail description of
this apparatus can be found in the text. (b) Quantum circuit for this
experiment, where the first qubit represents the photon and the second qubit
reveals the status of the bomb. (c) Our experiments on a range of IBM quantum
computers in 2020. The dashed-dotted line at 0.33 is the theoretical value
from Eq. 1. Each dot represents an execution of the experiment on the
corresponding quantum computer. The solid black lines are the average value of
dots. The device errors and standard deviations are 9.3%, 4.9%, 6.2%, 7.4%,
4.1%, 4.0%, 5.3% and 0.034, 0.047, 0.060, 0.029, 0.017, 0.025, 0.028, in order
given above, from left to right.

This experiment is described using a quantum circuit containing two Hadamard
gates together with a CNOT gate. In our setup as shown in Fig. 2 (b), the
first qubit represents the photon in the MZI, and the two Hadamard gates
simulate the two BSs. Like the bomb that detects the photon’s quantum state
inside the MZI, a CNOT gate between the two H gates is targeted on a second
qubit. The value of this second qubit $|q_{1}\rangle$ indicates if the bomb
has been touched or not. Let $|q_{1}\rangle=|1\rangle$ means the bomb is
touched, we summarize what happens in the circuit via a diagram as follows:

$H$$CNOT$$H$$|00\rangle$$|10\rangle$$|00\rangle$$|11\rangle$$|00\rangle$$|01\rangle$$|11\rangle$$|00\rangle$$|10\rangle$touched
andexploded bombno bomb or cannotdetect any bombdetected a bomb(50%)(25%)(25%)

Even though there is a $50\%$ probability of explosion, this scheme allows us
to probe the existence of a classical object in an interaction-free manner.
The efficiency rate $\eta$ can be expressed by a measurement on the two-qubit
system [33]

$\eta=\frac{P_{\text{det}}}{P_{\text{det}}+P_{\text{abs}}}=\frac{P_{|10\rangle}}{P_{|10\rangle}+P_{|01\rangle}+P_{|11\rangle}}=\frac{P_{|10\rangle}}{1-P_{|00\rangle}}.$
(1)

Here, $P_{\text{det}}$ is the probability of detecting the bomb without
touching it, and $P_{\text{abs}}$ is the probability that the bomb explodes;
or in other words, the photon is absorbed. With a 50:50 beamsplitter, the
efficiency rate $\eta=\frac{1/4}{1/4+1/2}=\frac{1}{3}$. We execute the quantum
circuit using a virtual quantum machine and obtain an exact value of 1/3,
shown as a dashed-dotted line in Fig. 2 (c). The same circuit running on real
IBMQ quantum hardware at various times in August 2020 yields results that
differ from each other and deviate from the theoretical value 1/3. We learn
that even when mitigation error correction [32] is applied, most systems have
large errors as shown with error bars in Fig. 2c, see Table 1 for more detail
for Essex, Ourense, Burlington, London, Vigo, Valencia, and x2. Especially,
IBM Essex is systematically off to one side, while IBM London is off to the
other side of the average value. Of all devices, the IBM Vigo and IBM Valencia
achieve the best result with the smallest errors in all aspects. We emphasize
that these results are obtained from an identical set of codes, and thus imply
large variations in IBM’s hardware. Even with a shallow circuit as shown in
Fig. 2b, one need to be cautious and choose a proper backend for their
calculations.

To qualify for “interaction-free measurement”[33], the efficiency $\eta$ needs
to reach 100%. In the optical setup, this efficiency can be obtained by a
series of connected MZIs such that the photon repeatedly detects the bomb
multiple times. In a setup with $N$ BSs, each with reflectivity
$R=\cos^{2}(\pi/2N)$, the photon gradually transfers from the lower to the
upper halves, as shown in Fig. 3 (a). After every beamsplitter, there is a
dominant chance that the photon takes the lower path and avoids the bomb, and
every time it does so the quantum state collapses and fully recovers at the
lower path before continuing to evolve unitarily [31, 38]. After $N-1$ stages
with $N$ beamsplitters, the probability that the photon only takes the lower
path becomes $[\cos^{2}(\pi/2N)]^{N}$. Following Eq. (1), the efficiency
$\eta$ [33] is

$\eta=\frac{\cos^{2N}(\frac{\pi}{2N})}{1-\sin^{2}(\frac{\pi}{2N})\cos^{2(N-1)}(\frac{\pi}{2N})}.$
(2)

The quantum circuit that simulates the series of MZIs in Fig. 3 (a) is shown
in Fig. 3 (b) for the case $N=5$, which is a natural expansion from the
circuit shown in Fig. 2 (b). The first qubit represents the chain of MZI and
the photon that pass through it. The Hadamard gate is now replaced by rotation
gates in the $y$-axis
$R_{y}(\theta)=\bigl{[}\begin{smallmatrix}\cos(\theta/2)&-\sin(\theta/2)\\\
\sin(\theta/2)&\cos(\theta/2)\end{smallmatrix}\bigr{]}$. We assume an
arbitrary angle for these rotations $R_{y,i}(\theta_{i})$, with $i\in[1,N]$.
These angles need to satisfy the condition

$\sum_{i=1}^{N}\theta_{i}=\pi,$ (3)

such that the measurement is performed on the original basis after all these
rotations. In contrast to Hadamard gates, a chain of $R_{y}(\theta_{i})$
satisfying Eq. (3) interferes $|0\rangle\rightarrow|1\rangle$, instead of
$|0\rangle\rightarrow|0\rangle$. To detect the status of the photon, a chain
of CNOT gates is added in between these rotation $R_{y}$. If any of these
target qubits change their value to $|1\rangle$, the photon is absorbed and
the bomb explodes. The state of interested is $|00..00\rangle$; it indicates
that the photon has successfully passed through the circuit without exploding
the bomb. Here, we post-select out the explosion cases, i.e., the target
qubits turn to $|1\rangle$. Since the CNOTs do not change the value of their
control qubit $q_{0}$, the coefficient for state $|00..00\rangle$ and
$|10..00\rangle$ are $\prod_{i=1}^{N}\cos(\theta_{i}/2)$ and
$\sin(\theta_{N}/2)\prod_{i=1}^{N-1}\cos(\theta_{i}/2)$, respectively. From
Eq. (1), $\eta=\frac{P_{\text{det}}}{P_{\text{det}}+P_{\text{abs}}}$, the
efficiency becomes

$\eta=\dfrac{P_{|00..00\rangle}}{1-P_{|10..00\rangle}}=\frac{\prod_{i=1}^{N}\cos^{2}\Big{(}\dfrac{\theta_{i}}{2}\Big{)}}{1-\sin^{2}\Big{(}\dfrac{\theta_{N}}{2}\Big{)}\prod_{i=1}^{N-1}\cos^{2}\Big{(}\dfrac{\theta_{i}}{2}\Big{)}}.$
(4)

It is easy to see that Eq. (2) is a special case of our result when all
$\theta_{i}$ are equal $\frac{\pi}{N}$.

Figure 3: The general Elitzur-Vaidman bomb experiment. (a) The original
optical setup [33] used a series of MZIs to detect the bomb multiple times.
(b) Quantum circuit for the experiment. The $R_{y}(\theta_{i})$ replaces the
$H$ gate for control over the reflectivity; an extension from Fig. 2b. (c) Our
experiment data from the IBMQ Valencia. Since Valencia is a 5-qubit computer,
the experiment extends to 4 stages. Device errors are 7.2%, 10.3%, 13.8%,
19.0% for stage $n=2,3,4,5$, respectively. Standard deviations for all points
are shown as error bars, with the largest at 0.087.

To simplify Eq. (4), we set $\theta_{N}=\theta$. Then from Eq. (3), we have
$\theta_{i}=\frac{\pi-\theta}{N-1}$ for $i\in[1,N-1]$. Following the result in
Fig. 2c, the circuit is executed on the Valencia quantum computer provided by
IBMQ [8, 22]. In Fig. 3 (c) we show theoretical curves from Eq. (4) in solid
gray, and simulations of the quantum circuit in Fig. 3 (b) using a virtual
machine as red dots. Experiment data from the Valencia device is shown as
circular dots in different colors for $N=2,3,4,5$, with device errors expected
at 7.2%, 10.3%, 13.8%, 19.0%, respectively. The code is executed 10 times,
each with 8192 shots, and error bars are calculated based on these deviations.
The measurement data fit pretty well to theory for a small number of BSs, at
$N=2$ and $3$. However, running the same quantum circuit on other systems,
such as London or Ourense, yields substantial errors, even with $N=2$ (data
not shown). At larger $N=$ 4 or 5, the experiment requires entanglement of
more qubits that demands a ”deeper” circuit and more gates. Our data are quite
off compared to theoretical and simulation results. This is not a surprising
result, as the IBM’s NISQ computers are known for noises and mostly limit to
shallow circuits [9, 10, 16, 17, 18, 19, 20, 21]. Finding the origin for these
errors is outside the scope of this work, as it requires in-depth information
from the chip design, fabrication process, microwave setup, cryostat, and
their inter-connections. Although being unreliable with quite many errors, it
is impressive that these computers reach $\eta>0.64$ for $N=3$ with $\theta$
around 0.6 $\pi$, and thus satisfactorily demonstrate the interaction-free
measurement. At the classical limit, we expect $\eta=0$, for such an effect
should be impossible.

## The Hardy’s paradox

The original Elitzur-Vaidman bomb can be extended to illustrate the Hardy’s
paradox [39, 40] with the optical setup showed in Fig. 4 (a). Here, the
classical bomb is replaced with two quantum objects, an electron and a
positron. They enter two MZIs arranged so that their path crosses, which would
lead to an annihilation typically. Without the positron, the electron would
always go to its constructive interference detector D1, and similarly for the
case of the positron that light up detector D4. A detection at the destructive
interference detectors D2 or D3 indicates that a counterpart is sent through
the MZI. The interesting case occurs when both D2 and D3 click, which
indicates that the electron and the positron collide at the crossing point.
According to their nature, they must annihilate and can not trigger D2,3
detectors. The coincident clicks are a paradox that a local hidden-variable
theory cannot explain [39]. Nevertheless, the Hardy’s paradox has been
theoretically explained in terms of weak values [41, 42, 43] and
experimentally verified [44, 45, 46]. Hardy’s paradox indicates the
nonlocality feature of the quantum state inside the two MZIs [47]. Nonlocality
is an intriguing topic that has been realized in photons [48, 49, 50, 51],
atoms [52, 53], or superconducting circuits [54, 55].

In Fig. 4 (b), the quantum circuit for this experiment is built similarly to
the two previous experiments. Two qubits represent the two MZIs with rotation
gates also satisfying Eq. (3) to put the qubits into their superpositions. In
between the rotations, a Toffoli gate is targeted on the third qubit,
revealing the quantum state of the control qubits when they are both at
$|1\rangle$. Finally, the qubits are rotated back to the original basis for
the measurement. With the original state denoted as
$|\psi\rangle=|000\rangle$, the final state after the quantum circuit is

$\displaystyle|\psi_{f}\rangle=-$
$\displaystyle\frac{1}{4}\sin\theta_{1}\sin\theta_{0}|000\rangle$
$\displaystyle+$
$\displaystyle\frac{1}{2}\sin\theta_{1}\sin^{2}\frac{\theta_{0}}{2}|100\rangle+\frac{1}{2}\sin^{2}\frac{\theta_{1}}{2}\sin\theta_{0}|010\rangle$
$\displaystyle+$
$\displaystyle\frac{1}{4}(2\cos\theta_{1}\sin^{2}\frac{\theta_{0}}{2}+\cos\theta_{0}+3)|110\rangle$
$\displaystyle+$
$\displaystyle\frac{1}{4}\sin\theta_{1}\sin{\theta_{0}}|001\rangle-\frac{1}{2}\sin^{2}\frac{\theta_{0}}{2}\sin{\theta_{1}}|101\rangle$
$\displaystyle-$
$\displaystyle\frac{1}{2}\sin^{2}{\frac{\theta_{1}}{2}}\sin{\theta_{0}}|011\rangle+\sin^{2}\frac{\theta_{1}}{2}\sin^{2}{\frac{\theta_{0}}{2}}|111\rangle.$
(5)

Here, $\theta_{i}$ is the rotation angle of the $i^{th}$ qubit, $i=[0,1]$. The
state of interest, called the nonlocal state, corresponds to the case when the
electron and the positron enter the cross-link without an annihilation. In our
setup using a $R_{y}(\theta_{i})$ and a $R_{y}(\pi-\theta_{i})$, it is state
$|000\rangle$ where the first two-qubit has value $|q_{0}=0,q_{1}=0\rangle$
and the last qubit is 0: $|q_{2}=0\rangle$. To resemble the electron-positron
annihilation in the quantum circuit,we reject all states with
$|q_{2}=1\rangle$ in Eq. (5) and obtain the remain probabilities from which
the nonlocal probability for $|000\rangle$ is

$\displaystyle\gamma=$
$\displaystyle\frac{P(|000\rangle)}{P(|000\rangle)+P(|100\rangle)+P(|010\rangle)+P(|110\rangle)}$
$\displaystyle=$
$\displaystyle\frac{\sin^{2}\theta_{1}\sin^{2}\theta_{0}}{4(2\cos\theta_{1}\sin^{2}\frac{\theta_{0}}{2}+\cos\theta_{0}+3)}$
(6) $\displaystyle=$
$\displaystyle\frac{2\sin^{4}\frac{\theta}{2}\cos^{2}\frac{\theta}{2}}{3-\cos\theta}.$
(7)

Figure 4: The Hardy’s paradox experiment. (a) The original optical setup [39]
where two MZIs are combined. (b) The quantum circuit, where each particle
passing through its BS is represented by a qubit evolves under the action of a
$R_{y}(\theta_{i})$ and a $R_{y}(\pi-\theta_{i})$ gate. The annihilation cases
are marked by the Toffoli gate. (c) Data from the IBM Vigo in direct
comparison with theory when $\theta=\theta_{0}=\theta_{1}$. (d) The nonlocal
probability as calculated from Eq. (6) showed in percentage. (e) Experiment
data for the nonlocal probability obtained from the IBMQ Vigo quantum computer
with a maximum of $\gamma=0.119$ at $\theta=0.533\pi$, close to Hardy’s result
[40]. The relative device error is estimated at 12.1% giving an absolute
device error of $0.014$, while the largest standard deviation is 0.015.

Equation (7) is obtained from Eq. (6) by choosing
$\theta_{1}=\theta_{0}=\theta$, i.e., the two MZIs are set up identically. To
show that our result is equivalent to Hardy’s original probability, we change
to new variables:
$\sin\frac{\theta}{2}=\frac{\sqrt{\alpha\beta}}{\sqrt{1-|\alpha\beta|}}$ and
$\cos\frac{\theta}{2}=\frac{|\alpha|-|\beta|}{\sqrt{1-|\alpha\beta|}}$. Eq.
(7) becomes

$\displaystyle\gamma=$
$\displaystyle-2\bigg{[}\frac{|\alpha\beta|(|\alpha|-|\beta|)}{1-|\alpha\beta|}\bigg{]}^{2}\frac{1}{(|\alpha|-|\beta|)^{2}+2|\alpha\beta|-3}$
$\displaystyle=$
$\displaystyle\bigg{[}\frac{|\alpha\beta|(|\alpha|-|\beta|)}{1-|\alpha\beta|}\bigg{]}^{2},$
(8)

which is Eq. 20 in Ref. [40]. The two approaches: the original calculation
[40] and the quantum circuit, are indeed equivalent, which yield a maximum
value $\gamma=\frac{1}{2}(5\sqrt{5}-11)\approx 0.090$ at $\theta=0.575\pi$. In
Fig. (4) (c), data from IBMQ Vigo are directly compared to theoretical result
Eq. (6), which show similar shapes of quite large error bars. These results
are fully plotted in 3D as a function of the two beamsplitters’ angles in Fig.
(4) (d) for theory and (e) for experiment. The maximum nonlocality probability
$\gamma$ measures at $0.119\pm 0.015$ around
$\theta_{0}=\theta_{1}=\theta=0.533\pi$, slightly larger than the value 0.090
estimated by Eq. 8 at $\theta_{0}=\theta_{1}=\theta=0.575\pi$. This is the
best result obtained by executing the quantum circuit in Fig. 4b multiple time
on various IBM’s quantum devices. Once again, the IBM quantum computer
reproduces the Hardy’s paradox reasonably well with decent precision,
especially with a Toffoli gate that demands complicated technical details.

## Discussion and Conclusion

Name date $T_{1}$ $T_{2}$ CNOT Readout Connectivity mm/yy ($\mu$s) ($\mu$s)
error (%) error (%) Burlington 08/20 84.88 67.36 1.50 4.64 T Essex 08/20
104.31 123.7 1.76 3.59 T London 08/20 61.45 62.74 1.75 4.40 T Ourense 08/20
93.15 66.43 0.92 2.96 T Valencia (Fig. 2) 08/20 84.18 62.78 1.11 2.32 T
Valencia (Fig. 3) 09/20 100.00 80.49 1.10 2.52 T Vigo (Fig. 1, 2) 08/20 73.28
50.73 1.07 1.66 T Vigo (Fig. 4) 09/20 107.64 74.04 0.94 1.96 T x2 08/20 57.08
45.40 1.82 3.18 $\hourglass$

Table 2: IBM quantum devices used in our experiments. ”Date” indicates the
time that we execute our codes, which can be found on Github [22]. All other
column including the lifetime $T_{1}$, decoherence time $T_{2}$, gate errors,
readout errors are recorded at the time of the experiments. Their values are
averaged over all 5 qubits. Connectivity denote the layout of the chip. Most
of them are designed as T shape, which mean the center qubit connects to three
other qubits.

The three chosen experiments all stemmed from the seeming non-definiteness of
quantum observables [30, 41, 56, 57]. Inside an MZI where the photon’s route
is uncertain, it is the premise for restoration of the interference patterns
in the eraser experiment, for the detection without interaction with the bomb,
and for the collision at the cross-link of the electron-positron pair without
an annihilation. As shown in Fig. 1 (a), Fig. 2 (a), Fig. 4 (a), these MZI
apparatus are quite similar, but a slight variation in the experiment setup
reveals profound quantum phenomena. Using an equivalent language, the quantum
circuits showed in Fig. 1 (b), Fig. 2 (b), Fig. 4 (b) are also quite similar.
Indeed, Fig. 1 (b) and Fig. 2 (b) are identical, implying a connection between
the quantum eraser experiment and the Elitzur-Vaidman bomb. To see the
similarity between the optical setup and a superconducting quantum processor,
we should think of photon travels inside the MZI as a function of time. In the
quantum circuit model, the quantum state evolves as a function of time under a
series of gate actions, as we read it from left to right. In the original
optical setup, these experiments explore the nonlocality of the photon path.
In the superconducting setup, there is no spatial separation between the two
eigenstates. Instead, the variable here is the charge number for charge
qubits.

In 2020, our experiments are performed on some of IBM’s quantum backends,
namely Athens, Burlington, Essex, London, Ourense, Valencia, Vigo, and x2.
These 5 qubit processors are of standard transmon design [7, 8] with lifetime
T1 and decoherence time T2 are in the range of 50-150 $\mu s$, see Table 2 for
more detail. They are equipped with an universal set of gates with fidelity in
the range of 99%. In their most popular layout, the center qubit in the
T-shape processors like Vigo or Valencia has the most connectivity: it is in
direct connection to three other ones. Although our quantum circuits are
compiled from standard gates such as H, CNOT, or $R_{y}$, they are translated
into the devices’ physical gates denoted U1, U2, and U3. As noise arise from
any possible sources, performance of these devices complicatedly depends on
the chip design, fabrication process, cryostat setup, which are not available
to end users. An in-depth analysis of these error is outside the scope of our
work, but we do employ certain techniques to manage error and achieve
reasonable results. Firstly, the fidelity of a circuit is estimated as the
product of fidelities of appliedgates given in that run, and the error of an
experiment can be deduced thereafter. Secondly, the quality are tightly
controlled as performances of these devices can fluctuate. Although calibrated
daily, the data can suddenly be bad. Thirdly, the best qubits of the best
machine are hand-picked for our circuit. Not all qubits are equal, and their
qualities vary as IBM constantly updates their system. Finally, we applied
mitigation error correction [32] extensively. Many times, this error-
correction scheme can vastly improve our results. In any case, IBM’s quantum
computers are state of the art devices that offer an unprecedented opportunity
to study the quantum world.

A central issue with gedanken quantum experiments is that they are difficult
to verify on physical systems. The three experiments in our work have been
realized on optical or atomic setups, the popular testbeds for foundation
quantum experiments. However, we are unaware of any work that reproduces them
on superconducting circuits, where the quantum object is the macroscopic
collective state of condensates electron pairs. Replicating these experiments
on the IBM quantum computers identifies two important aspects. First, it
benchmarks the quantum nature of the IBM machines. Our results show that all
experiments are reproduced with decent precision. The quantum eraser achieves
pretty high accuracy, while the original bomb experiments yield results that
fluctuate with devices, with the best machines being IBMQ Vigo and IBMQ
Valencia. However, the accuracy is lost when the quantum circuit contains more
than 4 qubits. Using the same codes and quantum circuit, we could not
reproduce the Elitzur-Vaidman bomb in the general case when the qubit number
is larger than 3. Apparently, we need gates with high fidelity, especially the
2 qubit gates that produce entanglement. Second, the IBM machines provide a
general-purpose testbed for quantum experiments. Remotely controlled using a
high-level language, we can reproduce a range of core quantum experiments in
one run. By submitting a single job to the IBM quantum cloud, we execute both
the quantum eraser, the Elitzur-Vaidman bomb, and the Hardy’s paradox at once
on a single solid-state device. More specially, we tune a wide range of
parameters and thus verify our calculations as shown in the general bomb
experiment in Fig. 3 (c). Similarly, in the Hardy’s paradox experiment, our
result spans the full range of $\theta_{0},\theta_{1}$, which require a
tremendous amount of work in the optical setup. Previously, such experiments
needed weeks for the demonstration. As shown in our calculations above, we
emphasize the equivalent between the quantum circuit language and the
traditional analysis.

In summary, we present quantum circuits for the quantum eraser, the Elitzur-
Vaidman bomb, and the Hardy’s paradox experiments and prove that they are
equivalent to the original setups using optical apparatuses. We further
execute them on the IBM quantum computers and obtain reasonable results. While
the current NISQ devices allow for the implementation of these experiments
with few qubits, it is reasonable to demonstrate these proof-of-principle
experiments as the testbed for testing quantum mechanics using noisy quantum
computers.

This work was supported by JSPS, Japan KAKENHI Grant Number 20F20021.

## References

  * [1] F. Arute et al., Quantum supremacy using a programmable superconducting processor, Nature 574, 505 (2019).
  * [2] H-S Zhong et al., Quantum computational advantage using photons, Science 370, 1460 (2020).
  * [3] D. P. DiVincenzo, The physical implementation of quantum computation, Fortschritte der Physik 48, 771 (2000).
  * [4] J. Preskill, Quantum Computing in the NISQ era and beyond, Quantum 2018-07-30.
  * [5] P. Krantz, M. Kjaergaard, F. Yan, T.P. Orlando, S. Gustavsson, W.D. Oliver, A quantum engineer’s guide to superconducting qubits, Applied Physics Reviews 6, 021318 (2019).
  * [6] Y. Nakamura, Yu. A. Pashkin, J. S. Tsai, Coherent control of macroscopic quantum states in a single-Cooper-pair box, Nature 398, 786 (1999).
  * [7] J. Koch, T.M. Yu, J. Gambetta, A.A. Houck, D.I. Schuster, J. Majer, A. Blais, M.H. Devoret, S.M. Girvin, R.J. Schoelkopf, Charge-insensitive qubit design derived from the Cooper pair box, Physical Review A 76, 042319 (2007).
  * [8] The IBM Quantum Experiences is electronically accessed at https://quantum-computing.ibm.com (2021).
  * [9] M. Amico, Z. H. Saleem, M. Kumph, Experimental study of Shor’s factoring algorithm using the IBM Q Experience, Physical Review A 100, 012305 (2019).
  * [10] A. Mandviwalla, K. Ohshiro, B. Ji, Implementing Grover’s algorithm on the IBM quantum computers, 2018 IEEE International Conference on Big Data (Big Data), 2531 (2018).
  * [11] J. Casanova, A. Mezzacapo, L. Lamata, E. Solano, Quantum simulation of interacting Fermion lattice models in trapped ions, Physical review letters 108, 190502 (2012).
  * [12] R. Barends, L. Lamata, J. Kelly, L. Garcıa-Alvarez, A.G. Fowler, A. Megrant, E. Jeffrey, T.C. White, D. Sank, J.Y. Mutus, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, I.C. Hoi, C. Neill, P.J.J. O’Malley, C. Quintana, P. Roushan, A. Vainsencher, J. Wenner, E. Solano, J. M. Martinis, Digital quantum simulation of fermionic models with a superconducting circuit, Nature Communications 6, 7654 (2015).
  * [13] C. A. Arguelles, B. J. P. Jones, Neutrino oscillations in a quantum processor, Physical review research 1, 033176 (2019).
  * [14] E. A. Martinez, C. A. Muschik, P. Schindler, D. Nigg, A. Erhard, M. Heyl, P. Hauke, M. Dalmonte, T. Monz, P. Zoller, R. Blatt, Real-time dynamics of lattice gauge theories with a few-qubit quantum computer, Nature 534, 516 (2016).
  * [15] M. Fitzpatrick, N. M. Sundaresan, A. C. Y. Li, J. Koch, A. A. Houck, Observation of a dissipative phase transition in a one-dimensional circuit QED lattice, Physical Review X 7, 011016 (2017).
  * [16] A. Kandala, A. Mezzacapo, K. Temme, M. Takita, M. Brink, J. M. Chow, J. M. Gambetta, Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets, Nature 549, 242 (2017).
  * [17] D. Garcia-Martin, G. Sierra, Five experimental tests on the 5-qubit IBM quantum computer, Journal of Applied Mathematics and Physics 6, 1460 (2018).
  * [18] M. Sisodia, Comparison the performance of five-qubit IBM quantum computers in terms of Bell states preparation, Quantum Information Processing 19, 214 (2020).
  * [19] N. Schwaller, M-A. Dupertuis, C. Javerzac-Galy, Evidence of the entanglement constraint on wave-particle duality using the IBM Q quantum computer, Physical Review A 103, 022409 (2021).
  * [20] H.-Y. Ku, N. Lambert, F.-J. Chan, C. Emary, Y.-N. Chen, F. Nori, Experimental test of non-macrorealistic cat states in the cloud, NPJ Quantum information 6, 98 (2020).
  * [21] S. J. Devitt, Performing quantum computing experiments in the cloud, Physical Review A 94, 032329 (2016).
  * [22] The code in this work can be accessed at: https://github.com/mx73/Testing-QM-on-NISQ
  * [23] X-S Ma, J. Kofler, A. Zeilinger, Delayed-choice gedanken experiments and their realizations, Reviews of Modern Physics 88, 015005 (2016).
  * [24] M. O. Scully, K. Drh̆l, Quantum eraser: A proposed photon correlation experiment concerning observation and “delayed choice” in quantum mechanics, Physical Review A 25, 2208 (1982).
  * [25] Y. H. Kim, R. Yu, S. P. Kulik, M. O. Scully, Y. Shih, Delayed “choice” quantum eraser, Physical Review Letters 84, 1 (2000).
  * [26] F. Kaiser, T. Coudreau, P. Milman, D.B.Ostrowsky, S. Tanzilli, Entanglement-enabled delayed-choice experiment, Science 338, 637 (2012).
  * [27] S. P. Walborn, M. O. Terra Cunha, S. Padua, C. H. Monken, Double-slit quantum eraser, Physical Review A 65, 033818 (2002).
  * [28] A. Bienfait, Y. P. Zhong, H.-S. Chang, M.-H. Chou, C. R. Conner, E. Dumur, J. Grebel, G. A. Peairs, R. G. Povey, K. J. Satzinger, A. N. Cleland, Quantum erasure using entangled surface acoustic phonons, Physical Review X 10, 021055 (2020).
  * [29] R. Ionicioiu, D.R. Terno, Proposal for a quantum delayed-choice experiment, Physical Review Letters 107, 230406 (2011).
  * [30] A. C. Elitzur, L. Vaidman, Quantum mechanical interaction-free measurements, Foundations of Physics 23, 987 (1993).
  * [31] L. Vaidman, The meaning of the interaction-free measurements, Foundations of Physics 33, 491 (2003).
  * [32] K. Temme, S. Bravyi, J. M. Gambetta, Error mitigation for short-depth quantum circuits, Physical Review Letters 119, 180509 (2017).
  * [33] P. Kwiat, H. Weinfurter, T. Herzog, M.A. Kasevich, A. Zeilinger, Interaction-free measurement, Physical Review Letters 74, 4763 (1995).
  * [34] A. E. Turner, C. W. Johnson, P. Kruit, B. J. McMorran, Interaction free measurement with electrons, Physical Review Letters 127, 110401 (2021).
  * [35] C. Robens, W. Alt, C. Emary, D. Meschede, A. Alberti, Atomic “bomb testing”: the Elitzur–Vaidman experiment violates the Leggett–Garg inequality, Applied Physics B 123, 11 (2017).
  * [36] L. Chirolli, E. Strambini, V. Giovannetti, F. Taddei, V. Piazza, R. Fazio, F. Beltram, G. Burkard, Electronic implementations of interaction-free measurements, Physical Review B 82, 045403 (2010).
  * [37] G.S. Paraoanu, Interaction free measurements with superconducting qubits, Physical Review Letters 97, 180406 (2006).
  * [38] Misra,B. and Sudarshan,E. C. G., The Zeno’s paradox in quantum theory, Journal of Mathematical Physics 18, 756 (1977).
  * [39] L. Hardy, Quantum mechanics, local realistic theories, and Lorentz-invariant realistic theories, Physical Review Letters 68, 2981 (1992).
  * [40] L. Hardy, Nonlocality for two particles without inequalities for almost all entangled states, Physical Review Letters 71, 1665 (1993).
  * [41] Y. Aharonov, A. Botero, S. Popescue, B. Reznik, J. Tollaksen, Revisiting Hardy’s paradox: counterfactual statements, real measurements, entanglement and weak values, Physics Letter A 301, 130-138 (2002).
  * [42] Y. Kedem and L. Vaidman, Modular values and weak values of quantum observables, Physical Review Letters 105, 230401 (2010).
  * [43] L. B. Ho, and N. Imoto, Full characterization of modular values for finite-dimensional systems, Physics Letter A 380, 2129-2135 (2016).
  * [44] W. Irvine, J. Hodelin, C. Simon, and D. Bouwmeester, Realization of Hardy’s thought experiment with photons, Physical Review Letters 95, 030401 (2005).
  * [45] J. S. Lundeen and A. M. Steinberg, Experimental joint weak measurement on a photon pair as a probe of Hardy’s Paradox, Physical Review Letters 102, 020404 (2009).
  * [46] K. Yokota, T. Yamamoto, M. Koashi, N. Imoto, Direct observation of Hardy’s paradox by joint weak measurement with an entangled photon pair, New Journal of Physics 11, 033011 (2009).
  * [47] N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani, S. Wehner, Bell nonlocality, Review of Modern Physics 86, 419 (2014).
  * [48] A. Aspect, J. Dalibard, G. Roger, Experimental test of Bell’s inequalities using time-varying analyzers, Physical Review Letters 49, 1804 (1982).
  * [49] A. Fedrizzi, R. Ursin, T. Herbst, M. Nespoli, R. Prevedel, T. Scheidl, F. Tiefenbacher, T. Jennewein, A. Zeilinger, High-fidelity transmission of entanglement over a high-loss free-space channel, Nature Physics 5, 389 (2009).
  * [50] A.G. White, D.F.V. James, P.H. Eberhard, P.G. Kwiat, Nonmaximally entangled states: production, characterization, and utilization, Physical Review Letters 83, 3103 (1999).
  * [51] Y.H. Luo, H.Y. Su, H.L. Huang, X.L. Wang, T. Yang, L. Li, N.L. Liu, J.L. Chen, C.Y. Lu, J.W. Pan, Experimental test of generalized Hardy’s paradox, Science Bulletin 63, 1611 (2018).
  * [52] D.N. Matsukevich, P. Maunz, D.L. Moehring, S. Olmschenk, C. Monroe, Bell Inequality violation with two remote atomic qubits, Physical Review Letters 100, 150404 (2008).
  * [53] J. Hofmann, M. Krug, N. Ortegel, L. Gerard, M. Weber, W. Rosenfeld, H. Weinfurter, Heralded entanglement between widely separated atoms, Science 337, 72 (2012).
  * [54] M. Neeley, R.C. Bialczak, M. Lenander, E. Lucero, M. Mariantoni, A.D. O’Connell, D. Sank, H. Wang, M. Weides, J. Wenner, Y. Yin, T. Yamamoto, A.N. Cleland, J.M. Martinis, Generation of three-qubit entangled states using superconducting phase qubits, Nature 467, 570 (2010).
  * [55] L. DiCarlo, M.D. Reed, L. Sun, B.R. Johnson, J.M. Chow, J.M. Gambetta, L. Frunzio, S.M. Girvin, M.H. Devoret, R.J. Schoelkopf, Preparation and measurement of three-qubit entanglement in a superconducting circuit, Nature 467, 574 (2010).
  * [56] Bell, J. S., On the Einstein Podolsky Rosen paradox, Physics Physique Fizika 1, 195-200 (1964).
  * [57] S. Kochen and E. P. Specker, The Problem of Hidden Variables in Quantum Mechanics, Journal of Mathematics and Mechanics 17, 59-87 (1967).