# 3D human tongue reconstruction from single “in-the-wild” images

Stylianos Ploumpis1,2 Stylianos Moschoglou1,2 0 Vasileios Triantafyllou2
Stefanos Zafeiriou1,2 Authors contributed equally. 1Imperial College London,
UK 2Huawei Technologies Co. Ltd
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

3D face reconstruction from a single image is a task that has garnered
increased interest in the Computer Vision community, especially due to its
broad use in a number of applications such as realistic 3D avatar creation,
pose invariant face recognition and face hallucination. Since the introduction
of the 3D Morphable Model in the late 90’s, we witnessed an explosion of
research aiming at particularly tackling this task. Nevertheless, despite the
increasing level of detail in the 3D face reconstructions from single images
mainly attributed to deep learning advances, finer and highly deformable
components of the face such as the tongue are still absent from all 3D face
models in the literature, although being very important for the realness of
the 3D avatar representations. In this work we present the first, to the best
of our knowledge, end-to-end trainable pipeline that accurately reconstructs
the 3D face together with the tongue. Moreover, we make this pipeline robust
in “in-the-wild” images by introducing a novel GAN method tailored for 3D
tongue surface generation. Finally, we make publicly available to the
community the first diverse tongue dataset, consisting of $1,800$ raw scans of
$700$ individuals varying in gender, age, and ethnicity backgrounds 111Project
url: https://github.com/steliosploumpis/3D_human_tongue_reconstruction. As we
demonstrate in an extensive series of quantitative as well as qualitative
experiments, our model proves to be robust and realistically captures the 3D
tongue structure, even in adverse “in-the-wild” conditions.

Figure 1: We propose a framework that accurately derives the 3D tongue shape
from single images. A high detailed 3D point cloud of the tongue surface and a
full head topology along with the tongue expression can be estimated from the
image domain. As we demonstrate, our framework is able to capture the tongue
shape even in adverse “in-the-wild” conditions.

##  1 Introduction

Recently, 3D face reconstruction from single “in-the-wild” images has been a
very active topic in Computer Vision with applications ranging from realistic
3D avatar creation to image imputation and face recognition [48, 17, 43, 25,
41, 15]. Nevertheless, despite the improvement in the quality of the 3D
reconstructions, all of these methods do not accommodate any statistical
variations in the oral cavity let alone a tongue template mesh. As a result,
the oral region is completely disregarded from the final result.

Being able to reconstruct the tongue expression has multiple advantages in
various applications. First of all, the generated avatars would be more
realistic and would be able to mimic many more facial expressions. Moreover,
speech animation tasks would be improved as the inclusion of the oral cavity
plays a significant role. Finally, face recognition applications could be
enhanced as more extreme poses and expressions would be modeled.

However, as we already pointed out, all of the current state-of-the-art (SOTA)
methods [48, 17, 43] do not contain the tongue component in their
implementations. This is because of two reasons: a) there is no publicly
available tongue dataset, and b) it is very challenging to carry out 3D
reconstruction of the face together with the tongue in “in-the-wild”
conditions, because of the highly deformable nature of the human tongue.

To tackle the absence of tongue data, we collected a large and diverse dataset
of textured 3D tongue point-clouds (more info about the data in Section 3).
Having captured the data, we created a pipeline which is comprised of the
following parts: a) a tongue point-cloud autoencoder (AE) which is used to
derive useful 3D features of our raw collected 3D data, b) a tongue image
encoder optimized based on the aforementioned 3D features, c) a shape decoder
which translates the encoder outputs to the parameter space of the Universal
Head Model (UHM) [37]. We should note that the UHM in our case is further
rigged/modified so that it can model various tongue shapes/expressions, as
explained in Section 3. We begin by training the AE in step a) and then we
train steps b-d) in an end-to-end fashion so that the output tongue expression
of the UHM model is as close as possible to the corresponding ground-truth 3D
tongue point-cloud of the 2D tongue image.

Since there is a lack of ground-truth 3D tongue data corresponding to “in-the-
wild” 2D tongue images, the pipeline we described so far is only trained using
our collected data which were captured under controlled conditions. This
results in sub-optimal performance in “in-the-wild” conditions. To remedy
this, we developed a novel conditional GAN framework that is able to generate
accurate 3D tongue point-clouds based on the image encoder outputs (step b) of
the pipeline). Having created new image/point-cloud pairs of “in-the-wild”
tongue data, we re-train the pipeline using also these new data. As we show in
Section 4, this addition substantially improves the quality of the tongue
reconstructions.

To summarize, the contributions of our work are the following:

  * •

We release a dataset of $1,800$ raw tongue scans of various shapes and
positions, corresponding to around $700$ subjects. Being the first such
diverse tongue dataset, it can be proven very useful to the community.

  * •

We present a complete pipeline trained in an end-to-end fashion that is able
to reconstruct the 3D face together with the tongue from a single image.

  * •

To make this pipeline robust to “in-the-wild” images, we introduce a novel GAN
framework which is able to accurately reconstruct 3D tongues from “in-the-
wild” images with an increasing level of detail.

Figure 2: Random 3D tongue expressions of our synthetic database based on the
mean UHM template. The expressions are rigged and manually sculpted to induce
more variance around the tongue surface and the general oral cavity.

##  2 Related Work

Single view 3D reconstruction of human facial/head parts is undeniably an
extremely valuable task in Computer Vision. However, it has posed many
challenges to the research community, due to the fundamental depth ambiguities
and the ill-posed nature of the problem. In order to constrain the ambiguity
of the problem, many statistical parametric models have been introduced for
different parts of the human face/head [3, 6, 29, 38].

Due to the increasing interest of facial analysis over the years the research
community has mainly focused on human facial reconstructions. Since the
inception of facial 3D Morphable Models (3DMMs) in [3], a myriad of scientific
papers have been published focusing solely on the reconstruction of facial
shape and appearance [4, 5, 17, 25]. Only recently with the emergence of 3D
scanning data has the research interest shifted to other significant parts of
the human head. A few head models have been introduced during the recent years
but without any statistical craniofacial correlations [29, 40]. The first
craniofacial 3DMM of the human head was introduced in [13] and later extended
and leveraged into a 3D head reconstruction setting from unconstrained single
images [38]. A few recent works tried to align a skull structure of the the
human head with the facial topology [30, 31] in order to obtain a distribution
of plausible face shapes given a skull shape.

Finer details of the human face/head started to appear with the introduction
of 3D human ear modeling [50]. Ears are key structures of the human head that
have an important contribution to the biometric recognition and general
appearance of a person. The two foremost examples of ear models were
introduced in [51, 12] but none of them was fused to a face/head in order to
create a complete appearance.

Moreover, in an attempt to overcome the “ _uncanny valley_ ” problem, a few
approaches have tried to model the independent variations/movements of the
human eye and the the facial eye region [46, 2]. These efforts are challenging
due to the limited amount of data around the eye region and the extreme level
of detail required for this task. Moving towards the oral cavity, teeth
modeling was introduced in [47, 44], where the 3D structure of the teeth was
recovered from 2D images via an elaborate optimization scheme.

Only very recently, a few approaches [28, 37] have tried to combine all of
these aforementioned attributes of the human head (eyes, ears, teeth, and
inner oral cavity) in order to build a complete model in terms of shape and
texture, which accurately represents the human head. Although these models
include an oral topology, none of them deals with the dynamics of the tongue,
something which is really important for speech animation and the overall
realness of the avatar representation. To this end, in this work we aim at
extending these approaches and paving the way towards a realistic human
appearance by releasing a diverse 3D tongue dataset to the research community.
We also present the first framework for accurate 3D human tongue
reconstruction from single images.

Figure 3: An illustration of our tongue reconstruction framework. First we
train the point-cloud AE on its own to get meaningful 3D features
($\mathbf{y}$) and then we trained the image encoder /shape decoder using a
number of different losses as explained in Section 3

##  3 3D human tongue reconstruction

In this Section we present the complete tongue reconstruction pipeline. We
begin by describing our collected 2D/3D tongue dataset and our manually rigged
tongue dataset which is based on the the UHM [37] template. We further provide
details about the point-cloud AE, the image encoder, the shape decoder and the
overall loss functions we used to optimize the pipeline for the tongue
reconstruction. Moreover, we present the novel conditional GAN method which is
able to accurately reconstruct 3D tongue point-clouds of “in-the-wild” tongue
images. Finally, we explain how we used the generated point-clouds of the GAN
to re-train the pipeline to achieve better results in “in-the-wild”
conditions.

###  3.1 Tongue datasets

TongueDB: the first 3D tongue dataset. As mentioned in Section 1, we collected
a large dataset comprising of textured 3D tongue scans. Our point cloud
database, dubbed TongueDB, contains approximately 1,800 3D tongue scans which
were captured during a special exhibition in the Science Museum, London. The
subjects were instructed to perform a range of tongue expressions (_i.e_.,
tongue out left and right, tongue out center, tongue out center round, tongue
out center extreme open mouth, tongue inside left and right, etc.). Some
example images can be seen in Fig. 6. The capturing apparatus utilized for
this task was a 3dMD 4 camera structured light stereo system, which produces
high quality dense meshes. We recorded a total of 700 distinct subjects with
available metadata about them, including their gender ($42\%$ male, $58\%$
female), age, and ethnicity ($82\%$ White, $9\%$ Asian, $3\%$ Black and $6\%$
other).

Rigged tongue database. In order to carry out 3D tongue and face
reconstruction, we would need to use a face/head model. Nevertheless, one
major drawback of all of the currently used face/head models [29, 40, 13] is
that they are missing the tongue component. This is because it is a
challenging task to non-rigidly capture in a fixed template the 3D topology of
the oral cavity. These challenges include: a) the highly deformable nature of
the tongue, b) the non-convexity of the mouth region, c) the specular texture
of the teeth. In order to alleviate this issue, we constructed a synthetic 3D
head and tongue dataset rigged by 3D artists. For our neutral mesh template
$\mathbf{\bar{T}}$ we utilize the mean template of the UHM [37] as it provides
all the necessary components of the human oral cavity in accordance with the
entire head statistical structure. The resulting rigged tongue expressions
rise at $75$ distinct meshes. In order to further augment our synthetic
dataset, we performed trilinear interpolation between the closest expression
meshes and generated a total of $n_{s}=720$ tongue expressions. Some example
synthetic expressions are shown in Fig. 2. A standard PCA was applied on the
interpolated meshes resulting in an orthogonal basis matrix
$\mathbf{U}_{t}\in\mathds{R}^{3N\times n_{t}}$ (where $N$ are the mesh
vertices and $n_{t}=110$ the kept components). The PCA is performed on the
entire set of head vertices and not solely on the oral cavity. In this way, it
is more efficient afterwards to transfer the tongue expression from the mean
head to a head with a different facial identity.

###  3.2 Method

Tongue point-cloud AE. In order to accurately reconstruct a tongue in its 3D
form based on a 2D image, our image encoder needs to be guided by meaningful
target labels which can capture all the desired 3D point-cloud information.
These labels, denoted as $\mathbf{y}\in\mathbb{R}^{256}$, are learned by
autoencoding the raw point-clouds of our dataset (_i.e_., the raw 3D tongue
scans). For this task, we utilize a self organizing-map framework for
hierarchical feature extraction [27].

Tongue image encoder. The task of the tongue image encoder is to produce
features which are close to the target 3D features $\mathbf{y}$ of the AE. To
make the encoder robust to various camera angles or illuminations, we employ a
rendering framework where we utilize the textured raw scans (TongueDB in
Section 3.1). We render our 1,8K textured meshes with a pre-computed radiance
transfer technique using spherical harmonics which efficiently represent
global light scattering. Additionally, we use more than 15 different indoor
scenes coupled with random light positions and mesh orientations around all 3D
axes, resulting in approximately 100K images. As an encoder we used a
ResNet-50 [20] model pre-trained on ImageNet [14] and fine-tuned it on our
dataset. In particular, we modified the last layer of the network to output a
vector $\tilde{\mathbf{y}}\in\mathbb{R}^{256}$ similar to the dimension of the
ground truth vector $\mathbf{y}$.

Shape decoder. In order to decode the encoder $\tilde{\mathbf{y}}$ labels into
meaningful tongue shapes, we use the synthetic PCA model $\mathbf{U}_{t}$ of
the rigged tongue expression dataset. To this end, after producing the
$\tilde{\mathbf{y}}$ labels, we utilize a standard multi-layer perceptron
(MLP) which works as a regression scheme to the latent parameters
$\mathbf{p}_{t}\in\mathds{R}^{110}$ of the synthetic PCA tongue model. The
statistical nature of the PCA model helps us constrain the final result during
training and ensures meaningful deformations which lie inside the spectrum of
our rigged/modified UHM model.

Pipeline training. During training, we first train the point-cloud AE on its
own and then train the pipeline of the image encoder and shape decoder in an
end-to-end fashion. To optimize the pipeline, we apply a total of $6$ losses
with each one contributing to the quality of the final result. The first $2$
losses are calculated between the predicted tongue expression of the
rigged/modified UHM model and the ground-truth tongue point-cloud of the
corresponding input image. Similarly to [45], we adopt a Chamfer loss [16]
$\mathcal{L}_{CD}$ to optimize the position of the resulting template points
as well as a normal loss $\mathcal{L}_{n}$ to correct the orientation of the
mesh. In order to compute an accurate Chamfer loss, we only utilize a small
area around the oral cavity which is defined based on the ground-truth point-
cloud. Additionally, we calculate a Laplacian regularization $\mathcal{L}_{l}$
loss between our predicted mesh the mean shape of the PCA model in order to
prevent the vertices from moving too freely outside the mean positions and
constrain the resulting shape to be smooth. An edge length loss
$\mathcal{L}_{e}$ is also introduced which penalizes any flying vertices
(outliers). Finally, we employ a collision loss $\mathcal{L}_{c}$ which
prevents the points of the tongue to penetrate the surface of the oral cavity
and is formulated as the sum of each collision error around the $12$ mouth
landmarks of the UHM template (as illustrated in the supplementary material):

$\mathcal{L}_{col}=\frac{1}{N}\sum_{k=0}^{11}\sum_{i=0}^{N-1}\max\left(0,d_{k}^{i}\right)$
(1)
$d_{k}^{i}=r^{2}-\left(q_{1}^{i}-x_{k}\right)^{2}-\left(q_{2}^{i}-y_{k}\right)^{2}-\left(q_{3}^{i}-z_{k}\right)^{2}$

Figure 4: Symbol $\mathbf{c}$ stands for row-wise concatenation along the
channel dimension. Symbol $\mathbf{o}$ stands for element-wise (_i.e_.,
Hadamard) product. The Generator inputs are a Gaussian noise sample
$\mathbf{z}$ and a label $\mathbf{y}$ corresponding to a particular tongue,
from which we want to sample a 3D point. The Discriminator input pairs are a
label $\mathbf{y}$ which corresponds to a specific tongue and $\mathbf{x}_{t}$
a real 3D point belonging to the aforementioned tongue point-cloud (but
sampled as explained in Section 3.3.1) and
$G\left(\mathbf{z},\mathbf{y}\right)=\tilde{\mathbf{x}_{t}}$ a generated point
belonging to this tongue. The Discriminator is asked to distinguish the real
from the fake point.

The $\mathcal{L}_{col}$ is calculated as the sum of distances of each collided
point $\mathbf{q}^{i}=\\{q_{1}^{i},q_{2}^{i},q_{3}^{i}\\}$ to the sphere $k$
with center the landmark coordinates $x_{k},y_{k},z_{k}$ and radius $r=1,5cm$.

Lastly, we impose a final L2 loss $\mathcal{L}_{y}$ in the intermediate step
of our pipeline where we constrain the predicted $\tilde{\mathbf{y}}$ encoded
features to be as close as possible to the ground-truth features
${\mathbf{y}}$ of the corresponding autoencoded point-cloud. This loss is of
paramount importance because: a) the $\tilde{\mathbf{y}}$ features in this way
contain rich 3D information invariant to texture/illumination variations and
b) our “in-the-wild” extension which we introduce later is based on a
generative point-cloud framework that relies on such rich 3D features.

The final loss function $\mathcal{L}_{total}$ is given by:

$\small{\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{CD}+\lambda_{2}\mathcal{L}_{n}+\lambda_{3}\mathcal{L}_{l}+\lambda_{4}\mathcal{L}_{e}+\lambda_{5}\mathcal{L}_{c}+\lambda_{6}\mathcal{L}_{y}}$
(2)

where $\lambda_{1},\ldots,\lambda_{6}$ are training hyper-parameters. During
inference, the encoder network takes as an input a single tongue image and
predicts a 3D embedding $\mathbf{\tilde{y}}$, which is later transformed to
the corresponding $\mathbf{p}_{t}$ parameters of the synthetic expression
model, through the MLP between the two latent spaces. Finally, we apply the
PCA model of the rigged head model on these $\mathbf{p}_{t}$ parameters to
derive the final mesh of the head with the tongue expression. An overview of
the methodology can be seen in Fig. 3.

###  3.3 TongueGAN for “in-the-wild” reconstruction

Although the pipeline presented in Section 3.2 provides a good estimation of
the tongue pose in the test set of our collected data, it does not perform
very well in “in-the-wild” images (Fig. 9). This behavior is expected because
our collected data were captured in controlled conditions and the training of
the encoder was carried out only with rendered images which do not fully mimic
“in-the-wild” conditions. To make our approach robust in “in-the-wild” images
too, we would need to further train the pipeline by using also such data.
However, for “in-the-wild” collected images from the web, we do not have their
corresponding 3D tongue point-clouds. As a result, to use “in-the-wild” data
in the pipeline, we would first need to have a method that can learn the
distribution of our collected 3D tongue data and generalize well.

Finding a method to generate novel 3D tongues is tricky. This is because of
several unique properties of the human tongue: a) it is a highly deformable
object, so we cannot register our collected data in a reference template and
apply relevant methods [7, 35, 39], b) it is a non-watertight surface (i.e.,
it contains holes) so we cannot also use any implicit function approximations
methods [36, 42, 33] or volumetric approaches [22, 23, 49]. Therefore, having
excluded the aforementioned categories, we decided to use GANs [18] for the 3D
tongue surface generations.

In order to generate accurate point-clouds that correspond to certain tongue
images, our GAN, dubbed as TongueGAN, needs to be guided by meaningful labels
which can capture all the desired 3D surface information. These labels are
provided by the trained point-cloud AE as described in Section 3. Since the
generation is driven by labels, TongueGAN is a conditional one [34].

In particular, given a label denoted as $\mathbf{y}$ and a random Gaussian
noise $\mathbf{z}\in\mathbb{R}^{128}$, the generator $G$ produces a novel
point-cloud point $G\left(\mathbf{z},\mathbf{y}\right)\in\mathbb{R}^{3}$,
which we denote as $\mathbf{\tilde{x}}_{t}$, that belongs to the tongue
surface represented by the label $\mathbf{y}$. On the other hand, the
discriminator $D$ receives as inputs the label $\mathbf{y}$, a real point-
cloud point $\mathbf{x}_{t}$ (which belongs to the tongue represented by the
label $\mathbf{y}$) and the generator output $\mathbf{\tilde{x}}_{t}$ and
tries to discriminate the fake (_i.e_., generated) from the real point. In the
mathematical parlance, this can be described as:

$\displaystyle\begin{array}[]{ll}\mathcal{L}_{D}=\mathbb{E}_{\mathbf{x}_{t}}\left[\log
D\left(\mathbf{x}_{t},\mathbf{y}\right)\right]-\mathbb{E}_{\mathbf{\tilde{x}}_{t}}\left[\log
D\left(\mathbf{\tilde{x}}_{t},\mathbf{y}\right)\right],\\\
\mathcal{L}_{G}=\mathbb{E}_{\mathbf{\tilde{x}}_{t}}\left[\log
D\left(\mathbf{\tilde{x}}_{t},\mathbf{y}\right)\right]\end{array}$ (5)

where $D$ tries to maximize $\mathcal{L}_{D}$, whereas $G$ tries to minimize
$\mathcal{L}_{G}$. The training process is considered complete when $D$ is no
longer able to differentiate between the real and fake point-cloud points.

Please note that instead of generating whole point-clouds for every provided
pair $(\mathbf{z},\mathbf{y})$ of noise and label, respectively, we merely
generate a point corresponding to the surface which the label $\mathbf{y}$
represents. That confers several advantages in comparison to the rest of the
methods in the literature, such as: a) we do not need to have in our training
set point-clouds with the same number of points and as a result we can train
our GAN without any data pre-processing on the raw point-clouds, which do not
have a fixed number of points among them, b) when it comes to generating
point-clouds corresponding to a particular label, we can generate on demand as
many points as we want and, contrary to the rest of the literature, we are not
constrained by any initially fixed resolution.

Since their inception, a plethora of GAN architectures and losses have been
introduced in the literature [1, 19, 8, 9, 32]. For the TongueGAN loss we
chose the Wasserstein loss with Gradient Penalty (WGAN with GP) [19] due to
its stability and good performance throughout the training process. As far as
the architecture is concerned, we turned our attention to the recently
proposed $\Pi$-Nets [11, 10], which are easy to implement and achieve state-
of-the-art results in a large battery of tasks, including graph representation
learning. In particular, we use our own, custom modification of $\Pi$-Nets to
accommodate the needs of our task. A graphical presentation of the network
structure is provided in Fig. 4.

Figure 5: A visual description of the mechanism explained in Section 3.3.1. In
a) we depict a raw tongue mesh with blue colour and with green colour we draw
a section. In b) we show the 3D points (green color) that belong to the tongue
surface and lie upon the section and with lighter green we depict the accepted
areas from which points can be sampled and then fed to the discriminator as
real data. These lighter green areas become gradually smaller as the training
evolves till they collapse to the ground-truth 3D points. In c) we provide a
zoomed-in area.

####  3.3.1 GAN loss for accurate surface approximation

Even though, as can be clearly seen in the experiments, our custom $\Pi$-Nets
modification along with the WGAN-GP loss significantly improve upon the
vanilla GAN or point-cloud GAN [26], there is still room for improvement as
the point-cloud representations are not ideally reconstructed (see Table 1).

We primarily attribute this to the strict behavior of the discriminator in
GANs (_i.e_., deciding in our case whether a generated point matches exactly a
point of the target point-cloud). This rigidity, especially in the early steps
of the training process, is not very helpful, as the generator struggles to
learn the real distribution of the point-clouds (_i.e_., all of the generated
points are discarded as fake by the discriminator with high confidence).

To remedy this, we slightly softened the discriminator, especially in the
initial steps, by slightly modifying the real points fed to it. To achieve
this, instead of directly feeding a real point $\mathbf{x}_{t}$ corresponding
to a label $\mathbf{y}$ to the discriminator, we feed the following:

$\displaystyle\mathbf{x}_{t^{\prime}|\mathbf{y}}\sim\mathcal{N}\left(\mathbf{x}_{t|\mathbf{y}},\sigma_{e}\mathbf{I}\right)$
(6)

where $\mathcal{N}\left(\mathbf{x}_{t|\mathbf{y}},\sigma_{e}\right)$ is a
multi-variate normal distribution with mean $\mathbf{x}_{t}$ and (isotropic)
variance $\sigma_{e}$. The variance $\sigma_{e}$ is not dependent on the label
$\mathbf{y}$. It is only dependent on the epoch $e$. By employing (6),
especially when the training process commences, the generator can better learn
the actual distribution as it does not get severely punished by the
discriminator when it slightly misses out the actual surface (see the
accompanying Fig. 5 for a visual understanding). As can be also seen in the
experiments, this addition yields better results and stabilizes the training
even further. We begin the training with a relatively small value for the
variance and further linearly reduce it as we go along with the training
process till it basically becomes zero towards the final epochs. This is
further corroborated empirically in Section 4.

####  3.3.2 Re-training the pipeline

After the training is complete, we use the trained generator together with the
trained encoder from Section 3.2. In this way, we create “in-the-wild” pairs
of 2D/3D data as follows: we feed the 2D “in-the-wild” image to the encoder
and get the label $\tilde{{\mathbf{y}}}$. We then use this label
$\tilde{{\mathbf{y}}}$ and the generator to produce a 3D point-cloud of the
input image. As we can see in Fig. 8, although TongueGAN is trained only on
our collected data, it is able to generalize very well in “in-the-wild” images
and as a result we can use it to create 2D/3D tongue pairs. We apply this
process to a number of “in-the-wild” images to create multiple pairs. Finally,
we re-train the pipeline we presented in Section 3.2, using also the
aforementioned pairs.

##  4 Experiments

This Section is organized as follows. We begin by providing details regarding
the training of the networks. Moreover, in Section 4.1, we outline a series of
quantitative as well as qualitative experiments under control conditions and
finally, in Section 4.2 we describe our results under “in-the-wild” images.

The MLP utilized for the regression between the labels $\tilde{\mathbf{y}}$
and the PCA parameters $\mathbf{p}_{t}$ in Section 3.2 has a structure of
(256, 128, 110) with a ReLU activation in the intermediate layers. The hyper
parameters of $\eqref{eq:total_loss}$ which balance the losses are
$\lambda_{1}=1.2$, $\lambda_{2}=1.6e-4$, $\lambda_{3}=0.4$, $\lambda_{4}=0.2$,
$\lambda_{5}=0.8$ and $\lambda_{6}=1.5$. As described in Section 3.3, for
TongueGAN we used a variant of WGAN with GP [19], which includes the injection
mechanism [11], as well as the surface loss function presented in Section
3.3.1. More specifically, we utilized a 9-layer Generator ($G$) and a 8-layer
Discriminator ($D$) with a total number of parameters of about $8\times
10^{6}$ and $4\times 10^{6}$, respectively. We trained TongueGAN using the
Adam optimizer [24] with $\left(\beta_{1}=0,\beta_{2}=0.9\right)$. We also
trained with a batch size of $2048$ for a total of $10^{6}$ iterations.
Following the idea introduced in [21], we use individual learning rates for D
and G with values of $1e-4$ and $1e-5$, respectively. Finally, we start
training with the variance $\sigma_{e}$ in $\eqref{eq:sampling}$ being $5e-3$
and we linearly decrease it by $10\%$ every $50\times 10^{3}$ steps. The exact
network structures are deferred to the supplementary material with more
details.

###  4.1 3D tongue reconstruction in control conditions

In this set of experiments, we used $90\%$ of TongueDB for training and the
rest for testing. Due to the intricacy of the tongue as a surface (as we
explained in detail in Section 3.3), we decided to use a GAN for the tongue
surface generation part. Moreover, an extra reason to utilize a GAN for the
training is the fact that it is able to generalize very well in unseen labels
during testing. To the best of our knowledge, the only method which has been
introduced in the literature and is able to carry out point-cloud generations
based on unseen labels is PointCloud GAN (PC-GAN) [26]. Consequently, in what
follows we draw comparisons against PC-GAN [26] and another two variants of
TongueGAN, namely: a) TongueGAN_v1, which is the same as TongueGAN with the
only difference being that the novel loss function (Section 3.3.1) is not
available in this version, and b) TongueGAN_v2, which is the typical GAN
structure where, instead of the injections we have simple concatenations along
the layers. Finally, we also report the results for the regressed tongue
expression (referred to as Tongue-Reg). For this, we only take into account a
small patch around the oral cavity defined by the ground truth point cloud, in
order to deduce a reasonable error.

Figure 6: Various raw 3D tongue scans of our database depicting different
tongue expressions along with the corresponding 2D renders. Figure 7:
Qualitative comparisons on various pointclouds from the test set of TongueDB.
a) input image, b) ground-truth point-cloud, c) the point-cloud generated by
TongueGAN and finally d) the point-cloud generated by point-cloud GAN [26].

Quantitative results are provided in Table 1 and qualitative results are
presented in Fig.7. For the quantitative results, we utilize the test set of
our TongueDB and we measure the error based on the two commonly used type of
distances when it comes to unordered 3D data, namely Chamfer Distance (CD) and
Earth Mover’s Distance (EMD) [16]. As can be clearly seen in all of the
comparisons, TongueGAN outperforms the compared methods by a large margin
whereas the regressed tongue expression outperforms the rest of the methods.

Figure 8: 3D head reconstructions with tongue animations from “in-the-wild” images. From left to right, we depict the “in-the-wild” image, then the point-cloud generations from two viewpoints and finally the 3D head reconstruction with a zoomed-in area around the oral cavity. Table 1: Quantitative comparisons among the compared methods using CD and EMD as metrics. Lower values indicate better performance. TongueGAN achieves the best results in all settings. _Method_ | _EMD_ | _CD_
---|---|---
TongueGAN | 1.62e-2 | 5.25e-5
Tongue-Reg | 1.79e-2 | 1.10e-4
PC-GAN | 1.82e-2 | 1.13e-4
TongueGAN_v1 | 1.97e-2 | 1.67e-4
TongueGAN_v2 | 2.24e-2 | 2.09e-4

###  4.2 3D tongue reconstruction “in-the-wild”

In this Section, we attempt to reconstruct the 3D surface of the tongue
together with the entire head structure from “in-the-wild” images. In this set
of experiments, we used all of TongueDB for training. We also added to our
training data another $5K$ “in-the-wild” tongue images and created their 3D
point-clouds using TongueGAN. Using all these data, we re-trained the pipeline
according to Section 3.3.2. The results are only visual as we do not have
ground-truth point-cloud data to report quantitative comparisons. Regarding
the comparisons, we should note that PointCloud GAN [26] cannot be used in
these experiments, as in order to work in the conditional setting, it needs as
input the actual ground-truth point-cloud it attempts to reconstruct,
something which we do not have at our disposal. Given that the TongueGAN
variations (_i.e_., TongueGAN_v1 and TongueGAN_v2) perform worse than
PointCloud GAN [26], we only present results in this Section regarding
TongueGAN.

Since our tongue regression method is based on the mean template mesh of UHM
we can easily utilize the pipeline presented in [37] in order to extent our
approach to a particular facial identity. We begin by fitting a facial mesh to
the image domain in order to get the 2D/3D landmarks and the identity of the
subject and then we regress to the full head topology based on the UHM model.
After reconstructing the head shape we crop the image around the projected 2D
mouth landmarks. We then feed this cropped image to the re-trained pipeline
and get the mean head shape with the tongue expression as mentioned in Section
3. Finally we merge the predicted tongue shape with the associated identity by
treating the predicted tongue expression as a separate blend-shape.

Some 3D reconstructions can be seen in Fig. 8. As evidenced, our pipeline is
able to accurately reconstruct the 3D tongue details even in “in-the-wild”
conditions. Additional tongue reconstructions of our method before and after
the re-training framework against state-of-the-art methods can be seen in Fig.
9. To further empirically validate that TongueGAN is able to capture the 3D
structures of random tongues that are not included in the training set we
provide linear interpolations between unseen latent features in the
supplementary material.

Figure 9: Qualitative shape evaluation between our approach and the state-of-
the-art methods of facial [17] (GANFit) and head [37] (UHM) reconstructions.
We can easily deduce that the re-training framework plays an important role in
the final tongue reconstruction from “in-the-wild” images.

##  5 Conclusion

In this work, we presented the first pipeline which is able to perform 3D head
and tongue reconstruction from a single image. To achieve this, we collected
the first diverse tongue dataset with various tongue shapes and positions
which we make publicly available to the research community. To also make this
pipeline robust in “in-the-wild” images and to mitigate the absence of their
corresponding ground-truth 3D tongue data, we introduced the first GAN method
that is tailored for accurately reconstructing the 3D surface of a tongue from
2D images. As we show in a series of experiments, we are now able to
accurately carry out 3D head reconstruction together with the tongue from a
single image and thus create more realistic 3D avatars.

## References

  * [1] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan.  arXiv preprint arXiv:1701.07875, 2017.
  * [2] Pascal Bérard, Derek Bradley, Markus Gross, and Thabo Beeler.  Lightweight eye capture using a parametric model.  ACM Transactions on Graphics (TOG), 35(4):1–12, 2016.
  * [3] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Proc 26th annual conf on Computer Graphics and Interactive Techniques, pages 187–194, 1999.
  * [4] James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou.  3d face morphable models” in-the-wild”.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5464–5473. IEEE, 2017.
  * [5] James Booth, Anastasios Roussos, Evangelos Ververas, Epameinondas Antonakos, Stylianos Ploumpis, Yannis Panagakis, and Stefanos Zafeiriou.  3d reconstruction of “in-the-wild” faces in images and videos.  IEEE transactions on pattern analysis and machine intelligence, 40(11):2638–2652, 2018.
  * [6] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway.  A 3d morphable model learnt from 10,000 faces.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5543–5552, 2016.
  * [7] Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael Bronstein, and Stefanos Zafeiriou.  Neural 3d morphable models: Spiral convolutional networks for 3d shape representation learning and generation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7213–7222, 2019.
  * [8] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  In Advances in neural information processing systems, pages 2172–2180, 2016.
  * [9] Shiyang Cheng, Michael Bronstein, Yuxiang Zhou, Irene Kotsia, Maja Pantic, and Stefanos Zafeiriou.  Meshgan: Non-linear 3d morphable models of faces.  arXiv preprint arXiv:1903.10384, 2019.
  * [10] Grigorios Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos Zafeiriou.  Deep polynomial neural networks.  arXiv preprint arXiv:2006.13026, 2020.
  * [11] Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng, and Stefanos Zafeiriou.  P-nets: Deep polynomial neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7325–7335, 2020.
  * [12] Hang Dai, Nick Pears, and William Smith.  A data-augmented 3d morphable model of the ear.  In Automatic Face & Gesture Recognition (FG 2018), 2018 13th IEEE International Conference on, pages 404–408. IEEE, 2018.
  * [13] Hang Dai, Nick Pears, William Smith, and Christian Duncan.  A 3d morphable model of craniofacial shape and texture variation.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 3104–3112, 2017.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [15] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.  3d morphable face models—past, present, and future.  ACM Transactions on Graphics (TOG), 39(5):1–38, 2020.
  * [16] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605–613, 2017.
  * [17] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou.  Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1155–1164, 2019.
  * [18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [19] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  In Advances in neural information processing systems, pages 5767–5777, 2017.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [22] Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos.  Large pose 3d face reconstruction from a single image via direct volumetric cnn regression.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1031–1039, 2017.
  * [23] Aaron S Jackson, Chris Manafas, and Georgios Tzimiropoulos.  3d human body reconstruction from a single image via volumetric regression.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0, 2018.
  * [24] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [25] Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis, Vasileios Triantafyllou, Abhijeet Ghosh, and Stefanos Zafeiriou.  Avatarme: Realistically renderable 3d facial reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 760–769, 2020.
  * [26] Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov.  Point cloud gan.  arXiv preprint arXiv:1810.05795, 2018.
  * [27] Jiaxin Li, Ben M Chen, and Gim Hee Lee.  So-net: Self-organizing network for point cloud analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9397–9406, 2018.
  * [28] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al.  Learning formation of physically-based face attributes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3410–3419, 2020.
  * [29] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194–1, 2017.
  * [30] Celong Liu and Xin Li.  Superimposition-guided facial reconstruction from skull.  arXiv preprint arXiv:1810.00107, 2018.
  * [31] Dennis Madsen, Marcel Lüthi, Andreas Schneider, and Thomas Vetter.  Probabilistic joint face-skull modelling for facial reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5295–5303, 2018.
  * [32] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  * [33] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Deep level sets: Implicit surface representations for 3d shape inference.  arXiv preprint arXiv:1901.06802, 2019.
  * [34] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [35] Stylianos Moschoglou, Stylianos Ploumpis, and Mihalis Nicolaou.  3dfacegan: Adversarial nets for 3d face representation, generation, and translation.
  * [36] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  arXiv preprint arXiv:1901.05103, 2019.
  * [37] Stylianos Ploumpis, Evangelos Ververas, Eimear O’Sullivan, Stylianos Moschoglou, Haoyang Wang, Nick Pears, William Smith, Baris Gecer, and Stefanos P Zafeiriou.  Towards a complete 3d morphable model of the human head.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [38] Stylianos Ploumpis, Haoyang Wang, Nick Pears, William AP Smith, and Stefanos Zafeiriou.  Combining 3d morphable models: A large scale face-and-head model.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10934–10943, 2019.
  * [39] Rolandos Alexandros Potamias, Jiali Zheng, Stylianos Ploumpis, Giorgos Bouritsas, Evangelos Ververas, and Stefanos Zafeiriou.  Learning to generate customized dynamic 3d facial expressions.  arXiv preprint arXiv:2007.09805, 2020.
  * [40] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black.  Generating 3d faces using convolutional mesh autoencoders.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 704–720, 2018.
  * [41] Nataniel Ruiz, Barry-John Theobald, Anurag Ranjan, Ahmed Hussein Abdelaziz, and Nicholas Apostoloff.  Morphgan: One-shot face synthesis gan for detecting recognition bias.  arXiv e-prints, pages arXiv–2012, 2020.
  * [42] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  arXiv preprint arXiv:1905.05172, 2019.
  * [43] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt.  Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2549–2559, 2018.
  * [44] Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novák, and Thabo Beeler.  Appearance capture and modeling of human teeth.  ACM Transactions on Graphics (TOG), 37(6):1–13, 2018.
  * [45] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 52–67, 2018.
  * [46] Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, and Andreas Bulling.  A 3d morphable eye region model for gaze estimation.  In European Conference on Computer Vision, pages 297–313, 2016\.
  * [47] Chenglei Wu, Derek Bradley, Pablo Garrido, Michael Zollhöfer, Christian Theobalt, Markus H Gross, and Thabo Beeler.  Model-based teeth reconstruction.  ACM Trans. Graph., 35(6):220–1, 2016.
  * [48] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1–10, 2020.
  * [49] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu.  Deephuman: 3d human reconstruction from a single image.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7739–7749, 2019.
  * [50] Yuxiang Zhou and Stefanos Zaferiou.  Deformable models of ears in-the-wild for alignment and recognition.  In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 626–633. IEEE, 2017.
  * [51] Reza Zolfaghari, Nicolas Epain, Craig T. Jin, Joan Glaunes, and Anthony Tew.  Generating a morphable model of ears.  pages 1771–1775. IEEE, Mar 2016.


revtex4-1Repair the float

# Photonic Implementation of the Quantum Morra Game

Andrés Ulibarrena  Institute of Photonics and Quantum Sciences, School of
Engineering and Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS,
United Kingdom  Alejandro Sopena  Instituto de Física Teorica, UAM-CSIC,
Universidad Autónoma de Madrid, Cantoblanco, Madrid, Spain  Russell Brooks
Institute of Photonics and Quantum Sciences, School of Engineering and
Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS, United Kingdom
Daniel Centeno  Instituto de Física Teorica, UAM-CSIC, Universidad Autónoma de
Madrid, Cantoblanco, Madrid, Spain  Perimeter Institute for Theoretical
Physics, Waterloo, Ontario, Canada, N2L 2Y5  Department of Physics and
Astronomy, University of Waterloo, Waterloo, Ontario, Canada, N2L 3G1  Joseph
Ho  Institute of Photonics and Quantum Sciences, School of Engineering and
Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS, United Kingdom
Germán Sierra  Instituto de Física Teorica, UAM-CSIC, Universidad Autónoma de
Madrid, Cantoblanco, Madrid, Spain  Alessandro Fedrizzi  Institute of
Photonics and Quantum Sciences, School of Engineering and Physical Sciences,
Heriot-Watt University, Edinburgh EH14 4AS, United Kingdom

###### Abstract

In this paper, we study a faithful translation of a two-player quantum Morra
game, which builds on previous work by including the classical game as a
special case. We propose a natural deformation of the game in the quantum
regime in which Alice has a winning advantage, breaking the balance of the
classical game. A Nash equilibrium can be found in some cases by employing a
pure strategy, which is impossible in the classical game where a mixed
strategy is always required. We prepared our states using photonic qubits on a
linear optics setup, with an average deviation $\leq 2\%$ with respect to the
measured outcome probabilities. Finally, we discuss potential applications of
the quantum Morra game to the study of quantum information and communication.

Modern game theory, initially developed by J. Nash and J. Neumann [1, 2] for
analysing economic strategies among rational players in a mathematically
consistent model, has found widespread applications in various fields,
including biology [3], politics [4], choice-theory [5] and computer science
[6]. However, the emergence of quantum information in the 80’s and 90’s
enabled researchers to study game theoretic models in the quantum regime and
it has turned out that superposition and entanglement allow new winning
strategies that previously did not exist [7, 8]. Quantum games (QG) have
already been used to design quantum cryptographic protocols with enhanced
security [9, 10] and optimise networks for frequency allocation [11].
Moreover, studying QG’s has revealed new insights on non-cooperative games
such as prisoners dilemma where a new equilibrium strategy was found and
improved the payoff to all players [12, 13]. Quantum games can now be
implemented on plethora of systems, such as small-scale quantum processors
[14], ion traps [15] or nitrogen vacancies in diamonds [16] however the most
common platform has been photonics [17, 18, 19, 20]. This has the benefit that
the games can be played between remote players, and that they can then be used
as primitives for higher level distributed quantum communication tasks [21].

Here we consider one of the oldest known games that is still studied today,
the Morra game [22, 23]. Morra is a non-cooperative game in which players hide
a maximum number of coins (or fingers), and each player attempts to guess the
total number. Players are ordered a priori, and the rule is that a player
cannot repeat the guess of the previous ones. This rule apparently gives the
first player an advantage over the rest, but unexpectedly everyone has an
equal chance of winning. The Morra is a closely related game to the Spanish
Chinos game, which has been studied extensively [24, 25]. The Morra’s game has
applications in modelling financial markets, and information transmission
[24].

Quantum versions of Morra’s game have been proposed in the past [26, 27], with
the aim of studying the equilibrium strategies between the two players. The
basic idea is to replace the act of drawing a coin by that of applying an
operator on a quantum state shared by all players, say a boson, or a hard-core
boson (fermions), a qubit, or two qubits. These operations create a state in
which one measures an observable that is the classical analogue of the total
number of coins. In some situations quantum effects may lead to the breaking
of the classical balance of the players.

Here we show a novel implementation of the quantum Morra game (QMG) using
qutrits that, unlike the previous proposals [26, 27], can reproduce the
classical game faithfully, and gives room for its deformation. This
implementation makes use of a three level system as shared resource for
players to act on, similar to the Aharonov three quantum boxes game [16]. We
implement this game employing entangled photonic qubits and obtained good
agreement between the theory and the experimental results.

We shall consider the simplest version of the Morra game that involves two
players, Alice and Bob, who can draw each from 0 to 1 coins and guess the
total number of coins with the restriction that Bob cannot repeat the result
predicted by Alice. This game can be generalised to more players and coins
[26]. We refer to a strategy as optimal when it leads to a Nash equilibrium
[2], and therefore both players’ strategies maximise both players’
preferences. A non-cooperative game which allows mixed strategies is
guaranteed to have at least one Nash equilibrium [28]. The optimal strategy
for Alice is to choose at random $c_{A}=0,1$ coins and to guess always
$g_{A}=1$, so as not to reveal information to Bob [26]. This is based on the
fact that with four possible outcomes, the most probable value of the sum is
1. Bob’s optimal strategy is to choose $c_{B}=0,1$ coins at random and make
his guess $g_{B}$ in a rational way [29]. If Bob chooses $c_{B}=0$, then by
reason he must exclude the option $g_{B}=2$, and if he chooses $c_{B}=1$, then
he must exclude $g_{B}=0$. Playing the optimal strategy, each player wins on
average half of the time, resulting in a symmetric game with equal winning
probabilities. This strategy is also Pareto optimal [30], since any change in
one player’s strategy would lead to an outcome less preferred by the other
player. This result is also valid for two players and a general number of
coins [26].

In our version of the QMG, first we will associate the total number of coins,
namely 0, 1, 2, to three orthogonal states
$|\tilde{0}\rangle,|\tilde{1}\rangle,|\tilde{2}\rangle$ that form the basis of
a qutrit. To produce these states, the players have two unitary operators,
${\cal O}_{0}$ and ${\cal O}_{1}$, that correspond to the number of coins they
have in their hands at the start of each roll. The joint state created by
Alice and Bob is given by

$|\psi_{a,b}\rangle={\cal O}_{a}^{A}{\cal O}_{b}^{B}|\phi\rangle,\qquad
a,b=0,1\,,$ (1)

where $|\phi\rangle$ is an initial state and note $|\psi_{0,1}\rangle$ and
$|\psi_{1,0}\rangle$ are identical states (a relative phase between them is
unobservable). Using this equation the basis states are obtained as
$|\tilde{0}\rangle=|\psi_{0,0}\rangle$,
$|\tilde{1}\rangle=|\psi_{0,1}\rangle=|\psi_{1,0}\rangle$ and
$|\tilde{2}\rangle=|\psi_{1,1}\rangle$. Choosing ${\cal O}_{0}$ as the
identity operator we find that $|\phi\rangle=|\tilde{0}\rangle$ and

$|\tilde{1}\rangle={\cal O}_{1}|\tilde{0}\rangle,\quad|\tilde{2}\rangle={\cal
O}_{1}^{2}|\tilde{0}\rangle\,.$ (2)

The operator ${\cal O}_{1}$ has to be unitary. A solution compatible with (2)
is ${\cal O}_{1}=X$, where $X$ is the Pauli matrix for qutrits,

$X=\left(\begin{array}[]{ccc}0&0&1\\\ 1&0&0\\\ 0&1&0\\\ \end{array}\right)\,,$
(3)

that satisfies $X^{3}=\mathbb{I}$. In the quantum game, the operator $X^{3}$
never arises because the maximum number of $X$ operators is 2. However, we
shall consider that having three coins in the box is the same as having none.

The outcome of a round of the game is determined by measuring the observable

$\hat{N}=\sum^{2}_{n=0}n|\tilde{n}\rangle\langle\tilde{n}|\ .$ (4)

The state after such measurement will be $|\tilde{0}\rangle$,
$|\tilde{1}\rangle$, or $|\tilde{2}\rangle$, thereby revealing the number of
coins. The winner of the round is the player whose guess matches the number of
coins.

In the current implementation, the number of outcomes scales with $(N\cdot
M)+1$ while in the previous ones it scaled with $(M+1)^{N}$, where $M$ is the
total number of coins per player and $N$ is the number of players. This is a
direct consequence of being able to trace back the origin of the coins to
their owners. As the number of coins and players involved grows, the
difference in the number of outcomes between the classical game and the
previous quantum implementations becomes increasingly larger.

We have translated the classical Morra game into a quantum game by means of a
one-to-one correspondence between classical and quantum objects. The quantum
realm allows us to define the superposition of coin states, in analogy to the
superposition of cat states. For this we use the quantum Fourier transform

${\cal
F}|\hat{j}\rangle=\frac{1}{\sqrt{3}}\sum_{k=0}^{2}\omega^{jk}|\tilde{k}\rangle,\quad\omega=e^{2i\pi/3}\,.$
(5)

Measuring the number of coins in each of the states (5) yields the values 0, 1
or 2 with equal probability $1/3$. Moreover, applying the operator $X$ to (5),
that is adding a quantum coin, just multiply these states by a global phase,
$X|\hat{j}\rangle=\omega^{-j}|\hat{j}\rangle\,.$

Along with the Pauli matrix $X$ for qutrits (3), there is a matrix $Z$ defined
as $Z=\operatorname{diag}(1,\omega,\omega^{2})$ that satisfies $ZX=\omega XZ$
and is related to $X$ by the Fourier transform

$X={\cal F}^{\dagger}Z{\cal F},$ (6)

where ${\cal
F}_{j,j^{\prime}}=\frac{1}{\sqrt{3}}\omega^{jj^{\prime}}\;(j,j^{\prime}=0,1,2)$.
This connection will allow us to deform the quantum game from the phase space.

The two players game can be deformed replacing $Z$ in (6) by
$Z_{\theta}=\textrm{diag}\left(1,e^{i\theta},e^{2i\theta}\right)$ yielding a
modified $X$ operator,

$X_{\theta}={\cal F}^{\dagger}Z_{\theta}{\cal
F}=\left(\begin{array}[]{ccc}x_{0}(\theta)&x_{2}(\theta)&x_{1}(\theta)\\\
x_{1}(\theta)&x_{0}(\theta)&x_{2}(\theta)\\\
x_{2}(\theta)&x_{1}(\theta)&x_{0}(\theta)\\\ \end{array}\right)\,,$ (7)

where

$x_{j}(\theta)=\frac{1}{3}(1+\omega^{2j}e^{i\theta}+\omega^{j}e^{2i\theta})\,.$
(8)

The states created by the action of $X_{\theta}$ are

$\displaystyle|1_{\theta}\rangle$
$\displaystyle=X_{\theta}|\tilde{0}\rangle=x_{0}(\theta)|\tilde{0}\rangle+x_{1}(\theta)|\tilde{1}\rangle+x_{2}(\theta)|\tilde{2}\rangle\,,$
(9) $\displaystyle|2_{\theta}\rangle$
$\displaystyle=X_{\theta}^{2}|\tilde{0}\rangle=x_{0}(2\theta)|\tilde{0}\rangle+x_{1}(2\theta)|\tilde{1}\rangle+x_{2}(2\theta)|\tilde{2}\rangle\,,$
(10)

where we used that $X_{\theta}^{2}=X_{2\theta}$. These states are normalised
but are not orthogonal except for $\theta=2\pi/3$ and $4\pi/3$. We have found
a general quantum circuit for the deformation unitary (7) using eight local
unitaries and three CNOT gates [31].

We recall that the probabilities are given by

$p_{a,b}(n)=|\langle n|\psi_{a,b}\rangle|^{2}$ (11)

where $|n\rangle$ are the states
$|\tilde{0}\rangle,|\tilde{1}\rangle,|\tilde{2}\rangle$ and
$|\psi_{a,b}\rangle$ are the common states generated by Alice and Bob using
the operators $X_{\theta}$ and $X_{\theta}^{2}$. In particular the average
probabilities of Alice are given by

$P_{a}(n)=\frac{1}{2}\sum_{b}p_{a,b}(n)=\frac{1}{2}(|x_{n}(a\theta)|^{2}+|x_{n}((a+1)\theta)|^{2})\
.$ (12)

We experimentally realise the QMG with a linear optics circuit, using
polarisation encoded photons to prepare the two-qubit state
$\ket{\psi_{a,b}}$. We transform our qutrits into a two qubit space, as
outlined in the supplement [31]. Let us denote by
$|i_{A}i_{B}\rangle,\;(i_{A},i_{B}=0,1)$ a basis of two qubits states, which
we encode in polarisation $(0,1\rightarrow H,V)$ and define our qutrit states
as

$\displaystyle|\tilde{0}\rangle$ $\displaystyle=$
$\displaystyle|H_{1}H_{2}\rangle\,,$ (13) $\displaystyle|\tilde{1}\rangle$
$\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}(|H_{1}V_{2}\rangle+|V_{1}H_{2}\rangle)\,,$
(14) $\displaystyle|\tilde{2}\rangle$ $\displaystyle=$
$\displaystyle|V_{1}V_{2}\rangle\,,$ (15)

allowing us to write the states $|1_{\theta}\rangle$ and $|2_{\theta}\rangle$
in the basis $|i_{1}i_{2}\rangle$, where the indices represents the photon at
measurement stage 1 and 2.

Figure 1 shows our experiment outlined in three stages; Alice and Bob input
their coin choice to the device, the device prepares the shared state, and
finally measures the outcome. We prepare our states (9) and (10) using the
parameterised half-wave plate configuration
$(\alpha_{1},\alpha_{2},\alpha_{3})$. We used solutions that match the desired
outcome probability of the shared state and preserve the entanglement entropy,
which we outline in the supplement [31]. These states require high fidelity
and purity to recreate the game faithfully, so we take advantage of a bright
photon-pair source using an aperiodically-poled potassium titanyl phosphate
(apKTP) crystal, which creates spectrally pure entangled photon pairs via
spontaneous parametric down-conversion (SPDC) [32, 33].

A winner of a single round of the game is decided by measuring the prepared
state in the Pauli-Z basis and revealing one of the outcomes (13-15).
Approximately 150,000 rounds are played for 34 evenly spaced values of
$\theta$ in the interval [0,$2\pi$]. The normalised probabilities for states
$|1_{\theta}\rangle$ and $|2_{\theta}\rangle$ are shown in the supplement
[31]. To analyse how these new outcome probabilities affect the players
strategies we have averaged Bob’s two plays over each of Alice’s choices using
(12), the results have been plotted in Figure 2a when Alice plays zero coins,
and Figure 2b when Alice plays one coin. The experimental values closely align
with the theoretical expectation, deviating by a maximum of 2% and confirms a
successful implementation of the QMG. We have measured a fidelity, compared to
the ideal states $|\tilde{0}\rangle$, $|\tilde{1}\rangle$ and
$|\tilde{2}\rangle$, of $97\%$. The main causes for the uncertainties in our
experimental data are due to factors such as imbalances on the PDC photon
generation loop, losses and depolarisation on the single-mode fibres and
imperfections in the retardance of the half waveplates.

Figure 1: The QMG experimental implementation. (a) Alice and Bob’s strategies are input into a device which prepares the joint state $\ket{\psi_{a,b}}$ and subsequently measures it. (b) Alice and Bob play the QMG by secretly depositing coins $c_{A}$ and $c_{B}$ into the device which has some pre-determined parameter $\theta$. Alice makes the first guess $g_{A}$, followed by Bob $g_{B}$. The device deforms the classical coins into quantum coins using the Unitary $X(\theta)$, performs a measurement and outputs the sum of the coins. (c) The linear optics circuit for state preparation begins with a photon pair source, constructed with a apKTP crystal housed in a Sagnac configuration [34, 35] pumped with a ti:sapphire laser centered at $775\text{\,}\mathrm{nm}$ and prepared in H with a linear polariser (POL), producing degenerate photon pairs at $1550\text{\,}\mathrm{nm}$, which are seperated from the pump with a dichroic mirror (DM). The states $\ket{\psi_{a,b}}$ are encoded using the three half-wave plates (HWP) $\alpha_{1},\alpha_{2},\alpha_{3}$, outlined in the supplement [31]. The measurement station consists of two polarising beam splitters (PBS) that project the two photons into the Z-basis, then coupled to superconducting nanowire single photon detectors (SNSPD), and 2-fold coincidences are identified using a time-tagging logic box. $\theta$ | $0$ | $2\pi/3$ | $\pi$ | $4\pi/3$
---|---|---|---|---
$P_{0}(0)$ | 0.99 | 0.50 | 0.57 | 0.51
$P_{0}(1)$ | $1.19\cdot 10^{-3}$ | 0.49 | 0.30 | $1.43\cdot 10^{-3}$
$P_{0}(2)$ | $7.63\cdot 10^{-3}$ | $2.46\cdot 10^{-3}$ | 0.13 | 0.49
$P_{1}(0)$ | 0.98 | $7.86\cdot 10^{-3}$ | 0.57 | $1.26\cdot 10^{-2}$
$P_{1}(1)$ | $2.46\cdot 10^{-3}$ | 0.50 | 0.30 | 0.49
$P_{1}(2)$ | $1.67\cdot 10^{-4}$ | 0.49 | 0.13 | 0.49
Table 1: Alice’s probability of winning ($P_{i}$) for different $\theta$
values, depending on the amount of coins in play ($i$).

In Table 1 we summarise the most notable points in the Figure 2, where
$P_{i}(G)$ is Alice’s probability of winning by guessing $G$ when playing $i$
coins.

At $\theta$ values close to $2\pi/3$, the classical regime is recovered,
matching the values predicted by the theory in Figure 2. Similarly, at
$4\pi/3$, the classical probabilities are reproduced, but the guesses for
coins 1 and 2 are flipped. This creates a classically impossible situation
since Alice plays no coins and can expect a 50% probability of winning by
guessing 2 coins.

Notably, for $\theta$ equalling 0 and $2\pi$, the game undergoes a complete
deformation, resulting in all coin states overlapping into the 0 coin outcome,
as we can see in Table 1. In this case, Alice will always win if she guesses
$0$ coins, regardless of the number of coins in play.

When $\theta=\pi$, the probable outcome for each coin is identical regardless
of the coin Alice picks. Even so, Alice still has an advantage over Bob as
guessing $|\tilde{0}\rangle$ yields a winning result higher than 0.5.

The deformation $\theta$ opens up the possibility for new strategies in which
Alice’s winning probability is higher than 50%. Each player’s strategy is
defined by the probability of playing zero or one coin, as well as their
guesses for each situation. We focus on analysing the best strategies for each
player when they play randomly, as well as characterising the optimal strategy
as a function of the deformation. Since Alice plays first, we consider
strategies in which her guess is independent of the number of coins she plays,
in order to not reveal information, similarly to the classical case.

(a)  (b)

Figure 2: Alice’s probability of winning versus $\theta$, for $T_{C}=1$ and
$T_{c}=2$. The probabilistic outcome of obtaining (0,1,2) coins when Alice
plays no coins ($\mathbb{I}$) LABEL:sub@fig:ProbsFigure1 and when she plays
one coin LABEL:sub@fig:ProbsFigure2 have been plotted on top of the
theoretical curves. Errors bars have calculated assuming Poisson statistics,
but found to be $\mathcal{O}(10^{-4})$, therefore neglected.

(a)  (b)  (c)

Figure 3: LABEL:sub@fig:Strategy_random The winning probabilities of Alice
(red) and Bob (green), as well as the probability of a draw (gray), when both
players choose 0 or 1 coin at random but make their best guess.
LABEL:sub@fig:Strategy_a Alice plays her best strategy knowing that Bob plays
randomly. LABEL:sub@fig:Strategy_b Bob plays his best strategy knowing that
Alice plays randomly. The experimental results have been plotted on top of the
theoretical values. Figure 4: Optimal strategies for Alice and Bob. The
winning probabilities for Alice (red) and Bob (green) are shown when each of
them plays with their best strategies. The probability of a draw, where
neither of them wins, is indicated in gray. The shaded area shows the values
of $\theta$ for which the strategies are pure. The experimental results have
been plotted on top of the theoretical values.

In the classical game, it is possible to achieve a Nash equilibrium with a
stable strategy where each player wins half the time by choosing 0 or 1 coins
at random. However, the deformation $\theta$ allows Alice to have an advantage
over Bob when playing randomly because she can make guesses that increase her
winning probability or the draw probability as shown in Figure 3a. In general,
these strategies are not stable, and both Alice and Bob can improve them.
Although Alice’s winning probability is higher than Bob’s, she can further
increase it, as shown in Figure 3b. Bob can try to improve his strategy as
well, but in some cases, he is unable to turn the situation around, and
Alice’s winning probability remains higher as shown in Figure 3c. In these
cases Alice’s strategy is stable, although not optimal.

Figure 3 allows us to verify if there are values of $\theta$ that allow an
optimal strategy when playing randomly apart from the classical case. This is
possible for $\theta=4\pi/3$ since it is equivalent to the classical case
under the exchange of states $|1_{\theta}\rangle$ and $|2_{\theta}\rangle$.
Remarkably, for $\theta=\pi$, it is also possible to achieve equilibrium by
playing randomly, although Alice’s winning probability is higher than Bob’s.
In this case, Alice must guess zero coins and Bob one or two. For other values
of theta, Alice and Bob’s improved strategies are pure.

We have calculated the winning probabilities for each player using the
experimental data for the overlaps, demonstrating a strong agreement with the
theory (see Figs. 4 and 3). One may wonder how the optimal strategy of each
player changes depending on the $\theta$ deformation. The winning
probabilities for each player when following the optimal and stable strategy
as a function of $\theta$ are shown in Figure 4. These strategies are mixed
for $\theta\in[4\pi/9,14\pi/9]$, while for the remaining values, they are
pure. The transition between mixed and pure strategies occurs at not trivial
$\theta$ values, these are determined by the construction of the operator
$X(\theta)$. Multiple applications of $X(\theta)$ will change the phase
relation between the states, which can be seen in the exponent of 8, therefore
this transition point is expected to be different with 3 players.

In contrast to the classical game, the deformation $\theta$ gives Alice an
advantage because she wins more often than Bob, and Bob cannot do anything to
change it. Another significant aspect that sets it apart from the classical
case is the possibility of both players losing. When playing optimally, the
classical game has been shown to be a zero-sum game. This is not true for the
quantum game and is more likely to occur at $\theta=\pi$ where there is a
$20\%$ chance that nobody wins. Moreover, the Nash equilibrium outside the
classical case is no longer Pareto-optimal except at $\theta=0,\ 4\pi/3$ and
$2\pi$.

The search for an optimum strategy using shared resources is closely aligned
with other competitive games, such the spectrum scarcity problem [11]. Our
analysis has shown that sometimes pure or mixed strategies are available, but
may come with different resource overheads. However, we did not consider these
resource costs in this paper and leave it as an open problem for future work.

A natural extension of the current implementation is to a three player game,
since the four possible outcomes can be encoded on just two photons. In fact,
the number of qubits scales as $\left\lceil{\log_{2}(M+1)}\right\rceil$ [31]
where $M$ represents the total number of coins, allowing for the efficient
construction of a network of players. Moreover, the selection of basis states
in this implementation remains arbitrary; we could have opted for three
distinct Bell pairs to encode $|\tilde{0}\rangle$, $|\tilde{1}\rangle$ and
$|\tilde{2}\rangle$. Using entangled states with minor modifications to the
game, could open the possibility of non-local strategies enabled by quantum
steering [36, 37]. This raises the question whether quantum games can be
exploited within the network context [21], that gain an advantage from multi-
partite resources [38, 35].

In summary, we have outlined a new construction of the two-player Quantum
Morra Game, providing Alice with a winning advantage over Bob that surpasses
the classical game. The deformation operator and theoretical states have been
realized within a linear optics setup, achieving high fidelity and low
standard deviation with respect to the measurement outcomes, which is less
than $2\%$.

This formulation of the Quantum Morra Game has bridged the classical and
quantum domains by incorporating the classical game as a particular instance
within the extended quantum game. In analysing the strategies, we have
identified a new Nash equilibrium that diverges from the classical game. This
has enhanced our understanding of strategies in quantum game theory and
provided insight into the realization of new quantum games. Surprisingly, it
remains unknown whether the equivalent of a Kakutani fixed-point theorem
exists in a complex space, such a discovery would guarantee a Nash equilibrium
for all quantum games [39]. Therefore, further theoretical development in
quantum game theory is needed to support the implementation of more complex
quantum games.

Finally, by extending the principles and techniques applied in this work to
other quantum games, we can potentially evaluate the robustness and efficiency
of quantum communication networks and deepen our understanding of quantum game
theory.

Acknowledgements This work was supported by the UK Engineering and Physical
Sciences Research Council (Grant Nos. EP/T001011/1.) A.S. is supported by the
Spanish Ministry of Science and Innovation under grant number
SEV-2016-0597-19-4. G.S. acknowledges the support of the Spanish Agencia
Estatal de Investigacion through the grants “IFT Centro de Excelencia Severo
Ochoa CEX2020-001007-S” and PID2021-127726NB-I00 funded by
MCIN/AEI/10.13039/501100011033 by ERDF, and the CSIC Research Platform on
Quantum Technologies PTI-001. DC was supported by Perimeter Institute for
Theoretical Physics. Research at Perimeter Institute is supported in part by
the Government of Canada through the Department of Innovation, Science and
Economic Development and by the Province of Ontario through the Ministry of
Colleges and Universities

Author contributions AS, DC and GS developed the theory, AU, RB and JH
performed the experiment. All authors contributed to the discussion of the
results and the writing of the manuscript.

Competing interests

The authors declare no competing interests.

## References

  * [1] Von Neumann, J. & Morgenstern, O.  _Theory of games and economic behavior, 2nd rev_ (Princeton university press, 1947).
  * [2] Nash, J. F.  Equilibrium points in n -person games.  _Proceedings of the National Academy of Sciences_ 36, 48–49 (1950).
  * [3] Ghirlanda, S. & Vallortigara, G.  The evolution of brain lateralization: a game-theoretical analysis of population structure.  _Proceedings of the Royal Society of London. Series B: Biological Sciences_ 271, 853–857 (2004).
  * [4] Snidal, D.  The Game Theory of International Politics.  _World Politics_ 38, 25–57 (1985).
  * [5] Luce, R. D. & Raiffa, H.  _Games and decisions: introduction and critical survey_ (Dover Publications, New York, 1989).
  * [6] Shoham, Y.  Computer science and game theory.  _Communications of the ACM_ 51, 74–79 (2008).
  * [7] Meyer, D. A.  Quantum Strategies.  _Physical Review Letters_ 82, 1052–1055 (1999).
  * [8] Ingarden, R. S.  Quantum information theory.  _Reports on Mathematical Physics_ 10, 43–72 (1976).
  * [9] Bennett, C. H. & Brassard, G.  Quantum cryptography: Public key distribution and coin tossing.  _Theoretical Computer Science_ 560, 7–11 (2014).
  * [10] Wiesner, S.  Conjugate coding.  _ACM SIGACT News_ 15, 78–88 (1983).
  * [11] Zabaleta, O. G., Barrangú, J. P. & Arizmendi, C. M.  Quantum game application to spectrum scarcity problems.  _Physica A: Statistical Mechanics and its Applications_ 466, 455–461 (2017).
  * [12] Eisert, J., Wilkens, M. & Lewenstein, M.  Quantum Games and Quantum Strategies.  _Physical Review Letters_ 83, 3077–3080 (1999).
  * [13] Benjamin, S. C. & Hayden, P. M.  Multiplayer quantum games.  _Physical Review A_ 64, 030301 (2001).
  * [14] Anand, A., Behera, B. K. & Panigrahi, P. K.  Solving diner’s dilemma game, circuit implementation and verification on the ibm quantum simulator.  _Quantum Information Processing_ 19, 1–14 (2020).
  * [15] Solmeyer, N. _et al._ Demonstration of a bayesian quantum game on an ion-trap quantum computer.  _Quantum Science and Technology_ 3, 045002 (2018).
  * [16] George, R. E. _et al._ Opening up three quantum boxes causes classically undetectable wavefunction collapse.  _Proceedings of the National Academy of Sciences_ 110, 3777–3781 (2013).
  * [17] Prevedel, R., Stefanov, A., Walther, P. & Zeilinger, A.  Experimental realization of a quantum game on a one-way quantum computer.  _New Journal of Physics_ 9, 205 (2007).
  * [18] Pinheiro, A. _et al._ Vector vortex implementation of a quantum game.  _JOSA B_ 30, 3210–3214 (2013).
  * [19] Schmid, C. _et al._ Experimental implementation of a four-player quantum game.  _New Journal of Physics_ 12, 063031 (2010).
  * [20] Balthazar, W. F., Passos, M. H. M., Schmidt, A. G. M., Caetano, D. P. & Huguenin, J. A. O.  Experimental realization of the quantum duel game using linear optical circuits.  _Journal of Physics B: Atomic, Molecular and Optical Physics_ 48, 165505 (2015).
  * [21] Dey, I., Marchetti, N., Caleffi, M. & Cacciapuoti, A. S.  Quantum game theory meets quantum networks.  _arXiv preprint arXiv:2306.08928_ (2023).
  * [22] Suetonius, S.  _The lives of the twelve Caesars_ (BoD–Books on Demand, 2019).
  * [23] Morris, P.  _Introduction to game theory_ (Springer Science & Business Media, 2012).
  * [24] Pastor-Abia, L., Pérez-Jordá, J. M., San-Fabián, E., Louis, E. & Vega-Redondo, F.  Strategic behavior and information transmission in a stylized (so-called chinos) guessing game.  _Advances in Complex Systems_ 04, 177–190 (2001).
  * [25] Pastor-Abia, L., San-Fabián, E., Louis, E. & Vega-Redondo, F.  Learning to play in a stylized (chinos) game: some preliminary results.  In _AIP Conference Proceedings_ , vol. 661, 167–173 (American Institute of Physics, 2003).
  * [26] Guinea, F. & Martín-Delgado, M. A.  Quantum Chinos game: winning strategies through quantum fluctuations.  _Journal of Physics A: Mathematical and General_ 36, L197–L204 (2003).
  * [27] Centeno, D. & Sierra, G.  General quantum chinos games.  _Journal of Physics Communications_ 6, 075009 (2022).
  * [28] Glicksberg, I. L.  A further generalization of the kakutani fixed theorem, with application to nash equilibrium points.  _Proceedings of the American Mathematical Society_ 3, 170–174 (1952).
  * [29] Bicchieri, C.  Rationality and game theory.  In Mele, A. R. & Rawling, P. (eds.) _The Oxford Handbook of Rationality_ , chap. 10, 182–205 (Oxford University Press, 2004), 1 edn.
  * [30] Pareto, V.  The Maximum of Utility Given by Free Competition.  _Giornale degli Economisti e Annali di Economia_ 67 (Anno 121), 387–403 (2008).
  * [31] Refer to the Supplemental Material for additional details on the experimental setup, the calculation of the waveplates parameters, the mapping from qutrits to qubits, the generalization of the deformed game, and the sources of noise.
  * [32] Graffitti, F., Barrow, P., Proietti, M., Kundys, D. & Fedrizzi, A.  Independent high-purity photons created in domain-engineered crystals.  _Optica_ 5, 514–517 (2018).
  * [33] Pickston, A. _et al._ Optimised domain-engineered crystals for pure telecom photon sources.  _Opt. Express_ 29, 6991–7002 (2021).
  * [34] Fedrizzi, A., Herbst, T., Poppe, A., Jennewein, T. & Zeilinger, A.  A wavelength-tunable fiber-coupled source of narrowband entangled photons.  _Optics Express_ 15, 15377–15386 (2007).
  * [35] Pickston, A. _et al._ Conference key agreement in a quantum network.  _npj Quantum Information_ 9, 82 (2023).
  * [36] Schrödinger, E.  Discussion of probability relations between separated systems.  In _Mathematical Proceedings of the Cambridge Philosophical Society_ , vol. 31, 555–563 (Cambridge University Press, 1935).
  * [37] Wiseman, H. M., Jones, S. J. & Doherty, A. C.  Steering, entanglement, nonlocality, and the einstein-podolsky-rosen paradox.  _Phys. Rev. Lett._ 98, 140402 (2007).
  * [38] Proietti, M. _et al._ Experimental quantum conference key agreement.  _Science Advances_ 7, eabe0395 (2021).
  * [39] Khan, F. S., Solmeyer, N., Balu, R. & Humble, T. S.  Quantum games: a review of the history, current state, and interpretation.  _Quantum Information Processing_ 17, 1–42 (2018).
  * [40] Byrd, R. H., Lu, P., Nocedal, J. & Zhu, C.  A Limited Memory Algorithm for Bound Constrained Optimization.  _SIAM Journal on Scientific Computing_ 16, 1190–1208 (1995).
  * [41] Virtanen, P. _et al._ SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ 17, 261–272 (2020).
  * [42] Vidal, G. & Dawson, C. M.  Universal quantum circuit for two-qubit transformations with three controlled-NOT gates.  _Physical Review A_ 69, 010301 (2004).
  * [43] Shende, V. V., Bullock, S. S. & Markov, I. L.  Recognizing small-circuit structure in two-qubit operators.  _Physical Review A_ 70, 012310 (2004).

Supplemental Material: Photonic Implementation of the Quantum Morra Game

This Supplemental Material is organised as follows: Section SM1 provides
additional details on the experimental setup as well as the calculation of the
waveplates parameters and the experimental overlaps. Section SM2 provides a
detailed description of the qutrit-to-qubit mapping required for the
experimental implementation of the game. Additionally, the unitary
transformation is decomposed into one-qubit gates and CNOT gates for
completenes. The deformed game is generalized to $N$ players and $M$ coins.

##  SM1 Methods

We prepare our states using a photon source housed in a Sagnac interferometer
to control the PDC process. Figure SM1 shows a Ti-Sapph laser pumping an
aperiodically-polled potassium titanyl phosphate (apKTP) crystal with 774.8nm
light, creating spectrally degenerate photons at 1549.6nm. A lens focuses the
beam onto the crystal for optimum PDC events and a Glan-Taylor (GT) prism
ensures only linearly polarised light enters the Sagnac while a Dichroic
Mirror (DM) reflects only down converted photons towards the coupler.

The first Half Wave Plate (HWP I) engineered for 775nm and the the Dual-
wavelength Polarisation Beam Splitter (DPBS) control the pump direction into
transmitted and reflected spatial modes around the Sagnac; Horizontal for
clockwise direction and Vertical for Anti-clockwise (AC). A dual-wavelength
HWP (DHWP) inside the Sagnc loop rotates V-pol light (In the reflected port)
into H-pol to facilitate PDC in the counter-propagating direction. When
pumping with D-pol light, the signal and idler photons are indistinguishable
and create a maximally entangled state (SM1).

$|\Psi^{+}\rangle=\frac{1}{\sqrt{2}}|h_{s}v_{i}\rangle+e^{i\phi}|v_{i}h_{s}\rangle$
(SM1)

We optimise the phase component from to the fibers and mirrors acting on the
two qubits using Polarisation Fiber Controllers (PFC) and recover the
$\Psi^{+}$. Down converted signal and idler photons exit the DPBS and are
rotated by two 1550nm HWP’s (II,III), parameterised by the numerical solution
to the Unitary (SM13) and coupled into a Single Mode Fiber (SFM).

Figure SM1: A Ti-Sapph laser pumps the apKTP crystal inside a Sagnac
inteferometer, producing photons pairs. A lens focuses the beam onto the
crystal for optimum PDC events and a Glan-Taylor (GT) ensures only linearly
polarised light enters the Sagnac loop. The first Half Wave Plate (HWP)
engineered for 775nm controls the pump direction of the Sagnac (Horizontal for
clockwise direction and Vertical for Anti-clockwise) once it passes through
the Dual-wavelength Polarisation Beam Splitter (DPBS) and the Dichroic Mirror
(DM) which reflects only the down converted photons. A dual-wavelength HWP
(DHWP) inside the Sagnc loop rotates light in the reflected port to facilitate
PDC with H polarised light. Down converted photons exit the DPBS and are
rotated by two 1550 HWP’s before being coupled into a Single Mode Fiber (SFM).
Polarisation Fiber Controllers (PFC) are used to apply the required unitary
operation for the encoded Bell states. Two measurement stations project the
photons into the Z-basis with a Polarising Beam Splitter (PBS) before being
coupled into SMF’s to Superconducting Nano-wire Single Photon Detectors
(SNSPD) and processed by a time-tagging logic box (a). The linear optics
circuit, (b) the encoding scheme for $|HH\rangle$ configuration when pumping
in Horizontal polarisation and waveplates {I,II,III} (c) Pumping in D.

### Waveplate parameters

The purpose of the experiment is to reproduce the probabilities $p_{a,b}(n)$
(11). Therefore, it is not necessary to reproduce the states
$\left|\psi_{a,b}\right\rangle$, but only to match the modulus of the
coefficients after projecting the state in the $Z$-basis,
$q_{i,j}=\left|\left\langle i,j|\psi_{a,b}\right\rangle\right|^{2}$ with
$i=0,1$ and $j=0,1$.

The experimental states $\left|\psi_{a,b}^{exp}\right\rangle$ produced by our
setup depend on three parameters,
$\vec{\alpha}=(\alpha_{1},\alpha_{2},\alpha_{3})$, through the half-waveplates
shown in Figure SM1. We need to find the parameters $\vec{\alpha}_{opt}$ that
best reproduce $q_{i,j}$. We simulate a noiseless version of the experimental
setup obtaining the parameterized states
$\left|\psi_{a,b}^{sim}(\vec{\alpha})\right\rangle$ and the overlaps
$q_{i,j}^{sim}(\vec{\alpha})$. This allows us to get $\vec{\alpha}_{opt}$ as
the solution to an optimization problem in which we aim to maximize the
Hellinger fidelity $F$ between $q_{i,j}$ and $q_{i,j}^{sim}(\vec{\alpha})$,

$F(\vec{\alpha})=\sum_{i,j}\sqrt{q_{i,j}q_{i,j}^{sim}(\vec{\alpha})}.$ (SM2)

In order to reproduce the entanglement of the system, we also require that the
entanglement entropies of the simulated states $S^{sim}(\vec{\alpha})$
reproduce the theoretical entropies $S$ (see Figure SM2c). Once again, this
can be formulated as an optimization problem where the objective is to
minimize $\left|S-S^{sim}(\vec{\alpha})\right|$. Taking into account this new
term, we seek to minimize

$C(\vec{\alpha})=1-F(\vec{\alpha})+\left|S-S^{sim}(\vec{\alpha})\right|,$
(SM3)

so that
$\vec{\alpha}_{opt}=\operatorname{argmin}\left(C(\vec{\alpha})\right)$. The
minimization is performed using the L-BFGS-B algorithm [40] implemented in
SciPy [41], reaching $C(\vec{\alpha}_{opt})<10^{-6}$ for all values of
$\theta$.

(a)  (b)  (c)

Figure SM2: Features of the states reproduced by our experimental setup. The
theoretical (solid line) and experimental (dots) overlaps $\left|\left\langle
n|1_{\theta}\right\rangle\right|^{2}$ and $\left|\left\langle
n|2_{\theta}\right\rangle\right|^{2}$ with $n=0\text{ (blue)},1\text{
(green)},2\text{ (orange)}$ are shown in LABEL:sub@fig:Prob_overlaps0 and
LABEL:sub@fig:Prob_overlaps1, respectively. The entanglement entropy of the
states $\left|1_{\theta}\right\rangle$ and $\left|2_{\theta}\right\rangle$ is
shown in LABEL:sub@fig:Entropies. The entanglement entropy calculated by
simulating the experimental setup without noise and using the optimal
parameters for the half-wave plates after numerical optimization is indicated
by dots.

### Supplementary Data

See Figure SM3 for the raw coincidence counts in the 4 sets of detector pairs.

(a)  (b)

Figure SM3: The coincidence counts on each of the 4 pairs of detectors coupled
to the Transmitted (T) and Reflected (R) ports of the PBS to reproduce the
overlaps for the states $\left|1_{\theta}\right\rangle$ LABEL:sub@fig:counts1
and $\left|2_{\theta}\right\rangle$ LABEL:sub@fig:counts2. Error bars have not
been plotted due to a low standard deviation at high counts.

##  SM2 Supplementary theory

### From qutrits to qubits

Let us denote by $|i_{A}i_{B}\rangle,\;(i_{A},i_{B}=0,1)$ a basis of two
qubits states. The qutrit states will be defined as

$\displaystyle|\tilde{0}\rangle$ $\displaystyle=$ $\displaystyle|00\rangle\,,$
(SM4) $\displaystyle|\tilde{1}\rangle$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}(|01\rangle+|10\rangle)\,,$ (SM5)
$\displaystyle|\tilde{2}\rangle$ $\displaystyle=$ $\displaystyle|11\rangle\,.$
(SM6)

Equation (3) expresses $X$ in the qutrit basis
$|I\rangle\;(I=\tilde{0},\tilde{1},\tilde{2})$. To implement the operation $X$
on two qubit states we have to extend the basis $|I\rangle$ to a fourth vector

$|\tilde{3}\rangle=\frac{1}{\sqrt{2}}(|01\rangle-|10\rangle)$ (SM7)

and promote $X\rightarrow X_{4}$ that acts according to (2) and trivially on
(SM7)

$X_{4}|\tilde{3}\rangle=|\tilde{3}\rangle\,.$ (SM8)

The matrix $X_{4}$ reads in the basis
$|I\rangle\;(I=\tilde{0},\tilde{1},\tilde{2},\tilde{3}$)

$X_{4}=\left(\begin{array}[]{cccc}0&0&1&0\\\ 1&0&0&0\\\ 0&1&0&0\\\ 0&0&0&1\\\
\end{array}\right)\,$ (SM9)

To find the action of $X_{4}$ on the two qubit states we use the unitary
transformation $V$ that expresses the basis $|I\rangle$ in terms of the basis
$|i_{A}i_{B}\rangle$

$|I\rangle=\sum_{i_{A}i_{B}}V_{i_{A}i_{B},I}|i_{A}i_{B}\rangle,\qquad
I=\tilde{0},\tilde{1},\tilde{2},\tilde{3}$ (SM10)

where

$V=\left(\begin{array}[]{cccc}1&0&0&0\\\
0&\frac{1}{\sqrt{2}}&0&\frac{1}{\sqrt{2}}\\\
0&\frac{1}{\sqrt{2}}&0&-\frac{1}{\sqrt{2}}\\\ 0&0&1&0\\\
\end{array}\right)\,.$ (SM11)

The rows are labelled by $00,01,10,11$ and the columns by
$\tilde{0},\tilde{1},\tilde{2},\tilde{3}$. The matrix expressing the action of
$X_{4}$ in the two qubit basis is given by

$X_{2\times 2}=VX_{4}V^{\dagger}=\left(\begin{array}[]{cccc}0&0&0&1\\\
\frac{1}{\sqrt{2}}&\frac{1}{{2}}&-\frac{1}{2}&0\\\
\frac{1}{\sqrt{2}}&-\frac{1}{{2}}&\frac{1}{2}&0\\\
0&\frac{1}{\sqrt{2}}&\frac{1}{\sqrt{2}}&0\\\ \end{array}\right)\ .$ (SM12)

Similarly, to find the action of the deformed unitary $X_{\theta}$ (7) on two
qubits we first let it act trivially on the state $|\tilde{3}\rangle$, as in
(SM8), and perform the change of basis (SM10) obtaining

$X_{\theta,2\times
2}=\left(\begin{array}[]{cccc}x_{0}(\theta)&\frac{1}{\sqrt{2}}x_{2}(\theta)&\frac{1}{\sqrt{2}}x_{2}(\theta)&x_{1}(\theta)\\\
\frac{1}{\sqrt{2}}x_{1}(\theta)&\frac{1}{2}(1+x_{0}(\theta))&\frac{1}{2}(-1+x_{0}(\theta))&\frac{1}{\sqrt{2}}x_{2}(\theta)\\\
\frac{1}{\sqrt{2}}x_{1}(\theta)&\frac{1}{2}(-1+x_{0}(\theta))&\frac{1}{2}(1+x_{0}(\theta))&\frac{1}{\sqrt{2}}x_{2}(\theta)\\\
x_{2}(\theta)&\frac{1}{\sqrt{2}}x_{1}(\theta)&\frac{1}{\sqrt{2}}x_{1}(\theta)&x_{0}(\theta)\end{array}\right)\,.$
(SM13)

This equation gives the states (10) in the two qubit basis

$\displaystyle|1_{\theta}\rangle$ $\displaystyle=$ $\displaystyle
x_{0}(\theta)|00\rangle+\frac{1}{\sqrt{2}}x_{1}(\theta)(|01\rangle+|10\rangle)+x_{2}(\theta)|11\rangle\,,$
(SM14) $\displaystyle|2_{\theta}\rangle$ $\displaystyle=$ $\displaystyle
x_{0}(2\theta)|00\rangle+\frac{1}{\sqrt{2}}x_{1}(2\theta)(|01\rangle+|10\rangle)+x_{2}(2\theta)|11\rangle\,.$

### Unitary decomposition

An arbitrary unitary $V\in U(4)$ can be implemented as a quantum circuit using
local gates and three CNOTs [42]. Furthermore, it is possible to use only two
CNOTs iff $\operatorname{tr}\left[V(\sigma_{y})^{\otimes
2}V^{t}(\sigma_{y})^{\otimes 2}\right]/\sqrt{\operatorname{det}(V)}$ is real
[43]. This condition is satisfied for the deformed unitary (SM13) when
$\theta=2\pi/3\ \text{(classical case)},\ 4\pi/3$. As indicated in the main
text, $\theta=4\pi/3$ reproduces the classical case under the interchange of
states $|1\rangle$ and $|2\rangle$. This means that the $\theta$-deformation
introduces an additional CNOT gate.

The circuits

$\quantikz&\gate{U(\beta_{2},\pi/4,-\pi/3)}\ctrl{1}\gate{R_{X}(-\pi/3)}\ctrl{1}\gate{U(\beta_{1},-2\pi/3,\pi/4)}\\\
\gate{U(\beta_{1},3\pi/4,2\pi/3)}\targ{}\gate{R_{Z}(\pi/3)}\targ{}\gate{U(\beta_{2},\pi/3,3\pi/4)},$
(SM15)
$\quantikz&\gate{U(\beta_{2},-\pi/4,-2\pi/3)}\ctrl{1}\gate{R_{X}(-\pi/3)}\ctrl{1}\gate{U(\beta_{1},-\pi/3,-\pi/4)}\\\
\gate{U(\beta_{1},-3\pi/4,\pi/3)}\targ{}\gate{R_{Z}(\pi/3)}\targ{}\gate{U(\beta_{2},2\pi/3,-3\pi/4)}$
(SM16)

with $\beta_{1}=\arccos(-1/3)/2$ and $\beta_{2}=\pi-\beta_{1}$ implement
(SM13) for $\theta=2\pi/3,\ 4\pi/3$, respectively. For the remaining values of
$\theta$, the circuits have the structure

$\quantikz&\gate{U(\vec{\alpha}_{1})}\ctrl{1}\gate{U(\vec{\alpha}_{3})}\ctrl{1}\gate{U(\vec{\alpha}_{5})}\ctrl{1}\gate{U(\vec{\alpha}_{7})}\\\
\gate{U(\vec{\alpha}_{2})}\targ{}\gate{U(\vec{\alpha}_{4})}\targ{}\gate{U(\vec{\alpha}_{6})}\targ{}\gate{U(\vec{\alpha}_{8})},$
(SM17)

and the parameters $\vec{\alpha}_{j}$ can be determined following the method
proposed in [42].

### General Quantum Morra Game

The game can be generalized to $n$ players and $m$ coins straightforwardly.
Given a pool with $r=nm$ coins, each player can draw from $0$ to $m$ coins and
guess the total number of coins, with the restriction that Bob cannot repeat
Alice’s guess. Similarly to the game with two players and one coin, we assign
a state $|\tilde{j}\rangle$ to each possible total number of coins
$j=0,1,\ldots,r$. In the classical case, each player has an operator $X_{r}$
that adds a coin when is applied to a state $\left|\tilde{j}\right\rangle$
with $j<r$, and $X_{r}^{r+1}=\mathbb{I}$. This operator is the
$r+1$-dimensional generalization of the Pauli matrix $\sigma_{1}$,

$X_{r}=\left(\begin{array}[]{cccccc}0&0&0&\cdots&0&1\\\ 1&0&0&\cdots&0&0\\\
0&1&0&\cdots&0&0\\\ 0&0&1&\cdots&0&0\\\
\vdots&\vdots&\vdots&\ddots&\vdots&\vdots\\\ 0&0&0&\cdots&1&0\\\
\end{array}\right)\,,$ (SM18)

and it is related to the operator
$Z_{r}=\operatorname{diag}(1,\omega,\ldots,\omega^{r})$ with
$\omega=e^{2\pi/(r+1)}$ by means of the Fourier transform ${\cal F}$,

$X_{r}={\cal F}^{\dagger}Z_{r}{\cal F}\,.$ (SM19)

Replacing $Z_{r}$ by $\operatorname{diag}(1,e^{i\theta},\ldots,e^{ri\theta})$
we get the deformed game and the modified $X_{r,\theta}$ operator which
creates the states

$|j_{\theta}\rangle=X^{j}_{r,\theta}|\tilde{0}\rangle=x_{0}(j\theta)|\tilde{0}\rangle+x_{1}(j\theta)|\tilde{1}\rangle+\ldots+x_{r}(j\theta)|\tilde{r}\rangle$
(SM20)

with

$x_{a}(\theta)=\frac{1}{r+1}\left(1+\sum_{k=1}^{r}\omega^{(r+1-k)a}e^{ki\theta}\right)\,.$
(SM21)

The generalized quantum Morra game can be implemented with
$\left\lceil{\log_{2}(r+1)}\right\rceil$ qubits.

### A Quantum Coin Deposit

It is well known that qubits belonging to a Bell pair are indistinguishable,
this is a hallmark of entanglement. An intriguing consequence of our choice of
encoding with a Bell pair results in the indistinguishably of the coins played
by either player. This may have application in anonymous, secure Quantum
Deposit system, wherein the choices of the participants remain concealed. This
system operates independently of classical communication channels, offering a
sharp departure from traditional games like Morra, which rely on some level of
communication. The Quantum Deposit model could lead to new security protocols
that require untraceable transactions or enhanced privacy. While the
exploration of these applications is beyond the scope of this paper, future
researchers could delve into the development of quantum cryptographic methods
that capitalize on this work.