# Some aspects of noise in binary classification with quantum circuits

Yonghoon Lee1, Dog̃a Murat Kürkçüog̃lu2, Gabriel N. Perdue2 1 The University
of Chicago, IL, 60637, US  2 Fermi National Accelerator Laboratory, Fermilab
Quantum Institute, PO Box 500, Batavia, IL, 60510-0500, USA

###### Abstract

We formally study the effects of a restricted single-qubit noise model
inspired by real quantum hardware, and corruption in quantum training data, on
the performance of binary classification using quantum circuits. We find that,
under the assumptions made in our noise model, that the measurement of a qubit
is affected only by the noises on that qubit even in the presence of
entanglement. Furthermore, when fitting a binary classifier using a quantum
dataset for training, we show that noise in the data can work as a
regularizer, implying potential benefits from the noise in certain cases for
machine learning problems.

FERMILAB-PUB-22-842-QIS current page.north east -3.5cm -1.5cm

Keywords— machine learning, quantum computing, quantum noise

##  1 Introduction

In the last decade there have been numerous advances in quantum computing and
machine learning. Advancements in quantum hardware have allowed for
substantial improvements in coherence times for qubits and gate fidelities and
we are drawing near to being able to construct and operate practical quantum
computers [6, 8, 2]. Indeed, there have already been demonstrations quantum
computers may offer advantages over other computing platforms for some
problems [1, 4]. Despite recent progress, the qubits and gate operations are
still strongly limited by noise, and decoherence effects are significant
problems that must be addressed [9].

Variational quantum classification (VQC) algorithms are among the most
important approaches in quantum machine learning (QML) [7]. As with all
quantum programs, the performance of VQCs is typically degraded when qubit and
gate operations are imperfect. In this work, we investigate the effect of
quantum noise in the binary classification task under a formally defined noise
model. While this noise model cannot perfectly describe the noise on a real
quantum computer, it shares many of the features found in real hardware. By
choosing a formally defined noise model we are able to analyze it rigorously
and provide foundations for future work under progressively more complex
constructions of the noise. The assumptions our noise model are a
simplification of the operating conditions of a real device and we do not
claim they fully capture all the important characteristics qubit noises.
Increasing the sophistication of the assumed noise model is a subject for
future investigation.

Our findings can be summarized in three theorems. In Theorem 1, we find that
under the assumption of only single-qubit noises in a quantum circuit, the
measurement at a qubit is affected only by the error in that qubit. This
result holds even if the circuit includes entangling gates before the errors
appear. In Theorem 2, we derive a closed form formula for the corrupted
measurement under Krauss and coherent errors. We find that quantum binary
classifier is robust to such noise in the sense that the binary output remains
the same unless the input value is sufficiently close to the classification
boundary. In Theorem 3, we study fitting a classifier using a quantum dataset,
and find that noise in the data can function as a regularizer, implying that
it can be beneficial in some cases. Proofs of all three theorems are provided
in the Appendix.

###  1.1 General notation

In this section, we will explain the notation used in this work. For a given
quantum state $\psi$, $\mathbb{P}_{\psi}\left\\{m^{j}=1\right\\}$ denotes the
probability to measure the qubit number $j$ in the excited state ($|1\rangle$)
defined by $\psi$ in the computational basis. For example, we write
$\mathbb{P}_{W\Phi(x)}\left\\{m^{1}=1\right\\}$ to denote the probability of
observing $|1\rangle$ from the first qubit of the output state when the input
$x$ is encoded by $\Phi$ and then is input to a quantum circuit defined by
$W$.

##  2 Effects of single qubit noises on the measurement of a quantum binary
classifier

In this section, we discuss how our noise model (defined by arbitrary unitary
single qubit noises) affects a variational quantum binary classifier.

###  2.1 Problem setting

Consider a quantum circuit semantically factorized into two pieces such that
the input $x$ passes through an encoder $\Phi$ (to load the classical input)
and then a parameterized circuit $W$. We can apply such a quantum circuit for
a binary classification task, where for simplicity we choose to base the
classifier on the readout of a single qubit.

To make this idea concrete, suppose we have two classes $\\{-1,1\\}$, and for
each $x$ in the input space $\mathcal{X}$, its class, denoted by $c(x)$, is
either $-1$ or $1$. Now suppose we are given a quantum binary classifier
$\hat{c}$, which classifies $x$ by

$\hat{c}(x)=\text{sign}\left(m(x)\right),$

where

$m(x)=\mathbb{P}_{W\Phi(x)}\\{m^{1}=1\\}-\frac{1}{2}=\langle\Phi(x)|W^{\dagger}M_{1}W|\Phi(x)\rangle-\frac{1}{2}$

denotes the probability of observing 1 at the first qubit. Here, $m^{1}$
denotes the measurement at the first qubit, and $M_{1}$ denotes the
corresponding measurement operator, given by

$M_{1}=\begin{bmatrix}0&0\\\ 0&1\end{bmatrix}\otimes I_{2}\otimes\cdots
I_{2}=\begin{bmatrix}\mbox{\normalsize 0}&\mbox{\normalsize 0}\\\
\mbox{\normalsize 0}&I_{2^{n-1}}\end{bmatrix}.$

Now we introduce noise model. We denote the corrupted circuit and encoder by
$\tilde{W}$ and $\tilde{\Phi}$, respectively. We assume the following:

$\begin{split}\tilde{W}&=UW,\\\ \tilde{\Phi}(x)&=V\Phi(x),\end{split}$ (1)

where

$\displaystyle U$ $\displaystyle=U_{1}\otimes U_{2}\otimes\cdots\otimes
U_{n},$ $\displaystyle V$ $\displaystyle=V_{1}\otimes
V_{2}\otimes\cdots\otimes V_{n},$

are unitaries with single qubit gates, which can be random. In other words,
our noise model assumes that an additional random gate appears for each of the
qubits and no additional entanglement occurs through the noise (there are no
two-qubit noises). For example, coherent error in a set of parameterized
single-qubit rotation gates would satisfy this condition.

###  2.2 Main results

Our first finding is that the output distribution of the measurement at a
qubit is not affected by the noises in the other qubits that occur after
entangling gates in the circuit.

######  Theorem 1.

Assume the noise model defined in Equation (1), and let
$U^{\prime}=U_{1}\otimes I_{2}\otimes\cdots\otimes I_{2}$ and
$V^{\prime}=V_{1}\otimes I_{2}\otimes\cdots\otimes I_{2}$.

  1. 1.

If there’s no encoder noise, i.e., $V\equiv I_{2^{n}}$, then the measurement
of the first qubit does not depend on $\\{U_{k}:2\leq k\leq n\\}$. In other
words,

$\mathbb{P}_{UW\Phi(x)}\\{m^{1}=1\\}=\mathbb{P}_{U^{\prime}W\Phi(x)}\\{m^{1}=1\\}$

holds for any input $x$ and any set of random unitary matrices
$U_{2},U_{3},\cdots,U_{n}\in\mathbb{C}^{2\times 2}$, where $m^{1}$ is a
shorthand notation for the measurement of the first qubit in the circuit.

  2. 2.

Moreover, if $W$ has no entangling gates, i.e., it has the form of

$W=W_{1}\otimes W_{2}\otimes\cdots\otimes W_{n},$

then

$\mathbb{P}_{UWV\Phi(x)}\\{m^{1}=1\\}=\mathbb{P}_{U^{\prime}WV^{\prime}\Phi(x)}\\{m^{1}=1\\}$

for any input $x$ and any set of random unitary matrices
$U_{2},U_{3},\cdots,U_{n}\in\mathbb{C}^{2\times 2}$ and
$V_{2},V_{3},\cdots,V_{n}\in\mathbb{C}^{2\times 2}$.

######  Remark 1.

In the proof of Theorem 1, we actually prove a more general result — the
statements hold for $U=U_{1}\otimes U_{2:n}$ and $V=V_{1}\otimes V_{2:n}$
where $U_{2:n}$ and $V_{2:n}$ are any unitaries with $n-1$ qubits. In other
words, for the invariance of the distribution of the measurement at the first
qubit, it is only required that there is no entangling noise that includes the
first qubit.

Figure 1:  The schematic for arrangement for Theorem 1. The state
$|\Phi(x)\rangle$ is prepared with some input $x$. (a) Then an arbitrary
unitary $W$ is applied and then qubit $1$ is measured, under the presence of
noise in $W$. Each $U_{i}$ is a random unitary which represents the noise
applied to qubit $i$. (b) The setting where $W$ has no entangling gates and we
additionally have the noise $V=V_{1}\otimes\cdots\otimes V_{n}$ in the
encoder.

Theorem 1 implies that no matter how many qubits we have, as long as we have
only the single-qubit noises, the performance of the quantum classifier is
affected only by the noise in the qubit we are measuring. In fact, this
statement implies that we can simply remove all the single-qubit gates in the
other qubits that appear after entanglements - it will change the quantum
circuit, but the resulting classifier remains the same.

We now begin to prepare the ground for a new theorem. We further consider the
second clause of Theorem 1 and set up a specific instance for the individual
noise terms to investigate the range of values that

$\tilde{m}(x)=\mathbb{P}_{UWV\Phi(x)}\\{m^{1}=1\\}-\frac{1}{2}$

can have.

For the noise in the encoder, it is sufficient to specify the model for
$V_{1}$, which acts only on qubit 1, using the result of Theorem 1:

$\begin{split}&V_{1}=\sigma_{C},\text{ where }\\\
&\mathbb{P}\\{C=j\\}=\begin{cases}1-3p&\text{ if }j=0\\\ p&\text{ if
}j=1,2,3\end{cases}.\end{split}$ (2)

Here, we let $\sigma_{0}=I_{2}$(i.e., no noise). In other words, we assume
that an additional $\sigma_{1}$, $\sigma_{2}$, or $\sigma_{3}$ gate randomly
appears, each with probability $p$, where $\sigma_{j}$ represents the standard
Pauli matrices:

$\sigma_{1}=\begin{bmatrix}0&1\\\
1&0\end{bmatrix},\sigma_{2}=\begin{bmatrix}0&-i\\\
i&0\end{bmatrix},\sigma_{3}=\begin{bmatrix}1&0\\\ 0&-1\end{bmatrix}.$

For the noise in $W$, again it is sufficient to set up a model for the noise
in the first qubit only. We assume

$\begin{split}&U_{1}=e^{-i\frac{\mu+\epsilon}{2}\sigma_{C^{\prime}}},\text{
where }\\\ &\mu\in[-\pi,\pi],\epsilon\sim N(0,\tau^{2}),\text{ and
}\mathbb{P}\\{{C^{\prime}=j}\\}=\begin{cases}1-3q&\text{ if }j=0\\\ q&\text{
if }j=1,2,3.\end{cases}.\end{split}$ (3)

Here, $N(0,\tau^{2})$ denotes the normal distribution with mean $0$ and
variance $\tau^{2}$. Under this noise model, we may prove the following
theorem:

######  Theorem 2.

If $W$ has no entangling gate, then under the noise model defined by Equations
(2) and (3), it holds that

$\tilde{m}(x)\in\eta\cdot[m(x)-\delta,m(x)+\delta],$

where

$\displaystyle\eta$ $\displaystyle=(1-4p)\cdot\left(1-2q(1-\cos\mu\cdot
e^{-(1/2)\tau^{2}})\right),$ $\displaystyle\delta$
$\displaystyle=\frac{2q\cdot|\sin\mu|\cdot e^{-(1/2)\tau^{2}}}{\eta}.$

Theorem 2 implies that as long as $p<1/4$ and $|m(x)|>\delta$, the sign of
$m(x)$ remains the same, i.e., $\text{sign}(\tilde{m}(x))=\text{sign}(m(x))$.
Note that $\lim_{\mu\rightarrow 0}\delta=\lim_{\tau\rightarrow 0}\delta=0$,
implying that the condition $|m(x)|>\delta$ tends to hold for moderate size of
noise. In other words, the corrupted binary classifier tends to provide the
same output as the original classifier, for $x$ values that are not too close
to the separating boundary.

However, Theorem 2 also implies that the noise can shrink $m(x)$. For example,
in the setting where $\mu=0$, we have
$\tilde{m}(x)=(1-4p)(1-2q(1-e^{-\frac{1}{2}\tau^{2}}))\cdot m(x)$. This
shrinkage of $m(x)$ implies that a larger sample of measurements would be
required for an accurate decision.

##  3 Effect of single qubit noises in the training data on the fitted
classifier

In this section, we study problem of training a binary classifier on a noisy
quantum dataset.

###  3.1 Problem setting

Suppose we have a “native” quantum dataset $(\psi_{i},Y_{i})_{1\leq i\leq
N}\subset\mathcal{H}\times\left\\{-1,1\right\\}$ where
$\mathcal{H}\in\mathbb{C}^{2^{n}}$, and $\psi_{i}$’s are allowed to be
entangled. Examples of a quantum dataset include outputs from quantum sensors
or a quantum computer. The distinction between an encoded classical dataset
and a naturally quantum dataset is not actually relevant here — the point is
we are not considering the encoding process if the dataset is classical.
Instead, we are beginning with quantum data that may be “corrupted” by noise.
The task is to fit a quantum circuit $W$ which classifies the points
accurately.

Here we consider empirical risk minimization to train the classifier. We use a
quantum circuit parametrized by $\theta$, and minimize the risk $R(\theta)$,
defined as

$R(\theta)=\mathbb{E}[\ell(m_{\theta}(\psi)\cdot Y)],$

where

$m_{\theta}(\psi)=\mathbb{P}_{W(\theta)\psi}\\{m^{1}=1\\}-\frac{1}{2}=\langle\psi|W(\theta)^{\dagger}M_{1}W(\theta)|\psi\rangle-\frac{1}{2}.$

Here the expectation is taken with respect to the distribution of $(\psi,Y)$
(which we assume that each $(\psi_{i},Y_{i})$ follows), and $\ell$ denotes the
loss function. The task is to fit a $W(\theta)$ of the form

$W(\theta)=W_{1}(\theta_{1})\times\cdots\times W_{n}(\theta_{n}),$

that accurately classifies input $\psi$’s. Note that we do not use entangling
gates in the classifier.

In practice, we approximate the risk by empirical risk $\hat{R}_{N}(\theta)$:

$\hat{R}_{N}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\ell(m_{\theta}(\psi_{i})\cdot
Y_{i}),$

and the estimator of $\theta$ is obtained by minimizing:

$\hat{\theta}_{N}=\operatorname*{argmin}_{\theta\in\mathbb{R}^{k}}\hat{R}_{N}(\theta)=\operatorname*{argmin}_{\theta\in\mathbb{R}^{k}}\frac{1}{N}\sum_{i=1}^{N}\ell(m_{\theta}(\psi_{i})\cdot
Y_{i}).$

Now suppose we observe a noisy dataset $\\{\tilde{\psi}_{i}\\}_{1\leq i\leq
N}=\\{V_{i}\psi_{i}\\}_{1\leq i\leq N}$ instead of
$\left\\{\psi_{i}\right\\}_{1\leq i\leq N}$. Then what we obtain by applying
the above procedure is the corrupted empirical risk

$\tilde{R}_{N}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\ell(m_{\theta}(\tilde{\psi}_{i})\cdot
Y_{i})$

where

$\tilde{m}_{\theta}(\psi_{i})=\langle\psi_{i}|V_{i}W(\theta)^{\dagger}M_{1}W(\theta)V_{i}|\psi_{i}\rangle-\frac{1}{2},$

and the corrupted estimator

$\tilde{\theta}_{N}=\operatorname*{argmin}_{\theta\in\mathbb{R}^{k}}\tilde{R}_{N}(\theta).$

We will look into the performance of $\tilde{\theta}_{N}$ and compare it to
the performance of the true estimator $\hat{\theta}_{N}$.

For the noise $V_{i}$, we consider the bitflip noise which we defined in
Equation (2):

######  Assumption 1.

The noise has the form of $V_{i}=V_{i}^{1}\otimes\cdots\otimes V_{i}^{n}$,
where

$V_{i}^{1}=\sigma_{C_{i}},i=1,2,\cdots,N,$

where $\\{C_{1},C_{2},\cdots,C_{N}\\}$ is an i.i.d sample from the
distribution of $C$ defined as the following.

$\mathbb{P}\\{C=j\\}=\begin{cases}1-3p&\text{ if }j=0\\\ p&\text{ if
}j=1,2,3\end{cases}.$

We assume the following for the loss function $\ell$:

######  Assumption 2.

The loss function $\ell:(-\frac{1}{2},\frac{1}{2})\mapsto[0,\infty)$ satisfies

  1. (a)

$\ell$ is convex and decreasing monotonically,

  2. (b)

$\ell(0)=1$.

Assumption 2 is satisfied by most well known loss functions, such as hinge
loss $\ell(t)=(1-t)_{+}$ and logistic loss $\log(1+\exp(-t))/\log(2)$.

###  3.2 Main results

Our main finding is that the noise in the data can work as a regularizer;
hence classifier performance can benefit from the noise in some cases. The
idea follows the argument of [3] which proves a similar result in the setting
where we fit a classifier with classical data $\\{X_{i},Y_{i}\\}_{1\leq i\leq
N}$.

######  Theorem 3.

Under Assumptions 1 and 2, the conditional expectation of the corrupted
empirical risk given the training data $\\{(\psi_{i},Y_{i})\\}_{1\leq i\leq
N}$ is given by

$\mathbb{E}\Big{[}\tilde{R}_{N}(\theta)\,\Big{|}\,\psi_{1:N},Y_{1:N}\Big{]}=(1-4p)\cdot\left[\hat{R}_{N}(\theta)+\lambda\cdot\hat{P}_{N}(\theta)\right],$

where $\hat{P}_{N}(\theta)$ satisfies

$\hat{P}_{N}(\theta)\geq\frac{1}{N}\sum_{i=1}^{N}\bigg{(}\frac{\ell\big{(}\frac{1}{3}|m_{\theta}(\psi_{i})|\big{)}+\ell\big{(}-\frac{1}{3}|m_{\theta}(\psi_{i})|\big{)}}{2}\bigg{)}.$

and $\lambda=\frac{4p}{1-4p}$.

Therefore, the corrupted estimator $\tilde{\theta}_{N}$ can be thought as an
approximation of the regularized estimator

$\theta_{N}^{regularized}=\operatorname*{argmin}_{\theta\in\mathbb{R}^{K}}\left[\hat{R}_{N}(\theta)+\lambda\cdot\hat{P}_{N}(\theta)\right]$
(4)

where $\lambda=4p/(1-4p)$. Note that under Assumption 2,
$t\mapsto\frac{\ell(\frac{1}{3}t)+\ell(-\frac{1}{3}t)}{2}$ is nondecreasing at
$t>0$, implying that having $\hat{P}_{N}(\theta)$ as a penalty term in (4)
results in a shrinkage on $|m_{\theta}(\psi_{i})|$ values. This prevents the
resulting estimator $\tilde{\theta}_{N}$ from excessively overfitting the
training data.

##  4 Conclusions

In this work, we studied the effects a noise model defined by single-qubit
errors on the performance of a binary classifier implemented with a quantum
circuit.

The most unexpected result from our finding is that even in the case of
entangling gates in a quantum circuit, the noise on a gate or qubit is only
going to affect the measurement from the specific qubit, i.e. the noise from
other qubits is not going to corrupt the measurement at the qubit of interest.

Our work also shows that the noise in the training data can even be beneficial
for the goal of finding an optimal quantum classifier. This is intuitive and
has been anecdotally observed before, even on real quantum hardware [5]. But
here we provide a formal argument supporting this phenomenon, although
conditioned on a particular noise model. These properties depend on the
structure of the problem, e.g., binary classification with the measurement at
the first qubit, which we discuss in this work.

Many questions yet remain. Will we still observe such robustness of the
measurement to the noise even in the case where we can have entangling gates
in the noise model, or cross-talk? These effects are known to be very
important for real quantum computers, and, indeed, two qubit gate performance
dominates concerns about single qubit gate performance on modern hardware
platforms. How will performance change in the setting where our goal is beyond
binary classification? We aim to explore these questions in future work.

##  5 Acknowledgements

This work was a project conducted while Y.L. was a student in the NSF MSGI and
NSF Graduate Research Fellowship Program under Grant No. DGE-1644869. D.K. and
G. P. were supported for this work by the DOE/HEP QuantISED program grant “HEP
Machine Learning and Optimization Go Quantum,” identification number
0000240323.

This document was prepared using the resources of the Fermi National
Accelerator Laboratory (Fermilab), a U.S. Department of Energy (DOE), Office
of Science, HEP User Facility. Fermilab is managed by Fermi Research Alliance,
LLC (FRA), acting under Contract No. DE-AC02-07CH11359.

## References

  * [1] Arute, Frank et. al.  Quantum supremacy using a programmable superconducting processor.  Nature, 574(7779):505–510, Oct 2019.
  * [2] Yu Chen, C Neill, Pedram Roushan, Nelson Leung, Michael Fang, Rami Barends, Julian Kelly, Brooks Campbell, Z Chen, Benjamin Chiaro, et al.  Qubit architecture with high coherence and fast tunable coupling.  Physical review letters, 113(22):220502, 2014.
  * [3] Yonghoon Lee and Rina Foygel Barber.  Binary classification with corrupted labels.  Electronic Journal of Statistics, 16(1):1367–1392, 2022.
  * [4] A. Morvan, B. Villalonga, X. Mi, S. Mandrà, A. Bengtsson, P. V. Klimov, Z. Chen, S. Hong, C. Erickson, I. K. Drozdov, J. Chau, G. Laun, R. Movassagh, A. Asfaw, L. T. A. N. Brandão, R. Peralta, D. Abanin, R. Acharya, R. Allen, T. I. Andersen, K. Anderson, M. Ansmann, F. Arute, K. Arya, J. Atalaya, J. C. Bardin, A. Bilmes, G. Bortoli, A. Bourassa, J. Bovaird, L. Brill, M. Broughton, B. B. Buckley, D. A. Buell, T. Burger, B. Burkett, N. Bushnell, J. Campero, H. S. Chang, B. Chiaro, D. Chik, C. Chou, J. Cogan, R. Collins, P. Conner, W. Courtney, A. L. Crook, B. Curtin, D. M. Debroy, A. Del Toro Barba, S. Demura, A. Di Paolo, A. Dunsworth, L. Faoro, E. Farhi, R. Fatemi, V. S. Ferreira, L. Flores Burgos, E. Forati, A. G. Fowler, B. Foxen, G. Garcia, E. Genois, W. Giang, C. Gidney, D. Gilboa, M. Giustina, R. Gosula, A. Grajales Dau, J. A. Gross, S. Habegger, M. C. Hamilton, M. Hansen, M. P. Harrigan, S. D. Harrington, P. Heu, M. R. Hoffmann, T. Huang, A. Huff, W. J. Huggins, L. B. Ioffe, S. V. Isakov, J. Iveland, E. Jeffrey, Z. Jiang, C. Jones, P. Juhas, D. Kafri, T. Khattar, M. Khezri, M. Kieferová, S. Kim, A. Kitaev, A. R. Klots, A. N. Korotkov, F. Kostritsa, J. M. Kreikebaum, D. Landhuis, P. Laptev, K. M. Lau, L. Laws, J. Lee, K. W. Lee, Y. D. Lensky, B. J. Lester, A. T. Lill, W. Liu, A. Locharla, F. D. Malone, O. Martin, S. Martin, J. R. McClean, M. McEwen, K. C. Miao, A. Mieszala, S. Montazeri, W. Mruczkiewicz, O. Naaman, M. Neeley, C. Neill, A. Nersisyan, M. Newman, J. H. Ng, A. Nguyen, M. Nguyen, M. Yuezhen Niu, T. E. O’Brien, S. Omonije, A. Opremcak, A. Petukhov, R. Potter, L. P. Pryadko, C. Quintana, D. M. Rhodes, C. Rocque, P. Roushan, N. C. Rubin, N. Saei, D. Sank, K. Sankaragomathi, K. J. Satzinger, H. F. Schurkus, C. Schuster, M. J. Shearn, A. Shorter, N. Shutty, V. Shvarts, V. Sivak, J. Skruzny, W. C. Smith, R. D. Somma, G. Sterling, D. Strain, M. Szalay, D. Thor, A. Torres, G. Vidal, C. Vollgraff Heidweiller, T. White, B. W. K. Woo, C. Xing, Z. J. Yao, P. Yeh, J. Yoo, G. Young, A. Zalcman, Y. Zhang, N. Zhu, N. Zobrist, E. G. Rieffel, R. Biswas, R. Babbush, D. Bacon, J. Hilton, E. Lucero, H. Neven, A. Megrant, J. Kelly, I. Aleiner, V. Smelyanskiy, K. Kechedzhi, Y. Chen, and S. Boixo.  Phase transition in random circuit sampling, 2023.
  * [5] Evan Peters, João Caldeira, Alan Ho, Stefan Leichenauer, Masoud Mohseni, Hartmut Neven, Panagiotis Spentzouris, Doug Strain, and Gabriel N. Perdue.  Machine learning of high dimensional data on a noisy quantum processor.  npj Quantum Information, 7(1):161, Nov 2021.
  * [6] Matthew Reagor, Wolfgang Pfaff, Christopher Axline, Reinier W Heeres, Nissim Ofek, Katrina Sliwa, Eric Holland, Chen Wang, Jacob Blumoff, Kevin Chou, et al.  Quantum memory with millisecond coherence in circuit qed.  Physical Review B, 94(1):014506, 2016.
  * [7] Maria Schuld, Alex Bocharov, Krysta M Svore, and Nathan Wiebe.  Circuit-centric quantum classifiers.  Physical Review A, 101(3):032308, 2020.
  * [8] Irfan Siddiqi.  Engineering high-coherence superconducting qubits.  Nature Reviews Materials, 6(10):875–891, 2021.
  * [9] Yiqing Zhou, E. Miles Stoudenmire, and Xavier Waintal.  What limits the simulation of quantum computers?  Phys. Rev. X, 10:041038, Nov 2020.

##  Appendix A Proof of Theorem 1

Let us write

$U_{2:n}=U_{2}\otimes\cdots\otimes U_{n}\text{ and
}U_{1}=\begin{bmatrix}u_{11}&u_{12}\\\ u_{21}&u_{22}\end{bmatrix},$

so that $U$ can be written as

$\displaystyle U=U_{1}\otimes U_{2:n}=\begin{bmatrix}u_{11}\cdot
U_{2:n}&u_{12}\cdot U_{2:n}\\\ u_{21}\cdot U_{2:n}&u_{22}\cdot
U_{2:n}\end{bmatrix},$

and write

$\displaystyle\Phi(x)=\begin{bmatrix}\Phi_{1}(x)\\\ \Phi_{2}(x)\end{bmatrix}.$

Then we have

$\displaystyle\mathbb{P}_{UW\Phi(x)}\left\\{m^{1}=1\right\\}$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[\mathbb{P}_{UW\Phi(x)}\left\\{m^{1}=1\ |\
U\right\\}\right]$ $\displaystyle=$
$\displaystyle\mathbb{E}\big{[}\langle\Phi(x)|W^{\dagger}U^{\dagger}M_{1}UW|\Phi(x)\rangle\big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Bigg{[}\Bigg{\|}\begin{bmatrix}\mbox{\normalsize
0}&\mbox{\normalsize 0}\\\ \mbox{\normalsize
0}&I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}u_{11}\cdot U_{2:n}&u_{12}\cdot
U_{2:n}\\\ u_{21}\cdot U_{2:n}&u_{22}\cdot
U_{2:n}\end{bmatrix}\begin{bmatrix}\Phi_{1}(x)\\\
\Phi_{2}(x)\end{bmatrix}\Bigg{\|}^{2}\Bigg{]}$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[\Big{\|}{u_{21}\cdot
U_{2:n}\Phi_{1}(x)+u_{22}\cdot U_{2:n}\Phi_{2}(x)}\Big{\|}^{2}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}\left[\Big{\|}{u_{21}\cdot\Phi_{1}(x)+u_{22}\cdot\Phi_{2}(x)}\Big{\|}^{2}\right],\textnormal{\quad
since $U_{2:n}$ is unitary}$

which does not depend on the distribution of $U_{2:n}$. Hence, the first claim
is proved.

Similarly, for the second claim we write

$W_{2:n}=W_{2}\otimes\cdots\otimes W_{n}\text{ and
}W_{1}=\begin{bmatrix}w_{11}&w_{12}\\\ w_{21}&w_{22}\end{bmatrix},$

and

$V_{2:n}=V_{2}\otimes\cdots\otimes V_{n}\text{ and
}V_{1}=\begin{bmatrix}v_{11}&v_{12}\\\ v_{21}&v_{22}\end{bmatrix},$

then it holds that

$\begin{split}&\mathbb{P}_{UWV\Phi(x)}\left\\{m^{1}=1\right\\}\\\
=&\mathbb{E}\left[\mathbb{P}_{UWV\Phi(x)}\left\\{m^{1}=1\ |\
U,V\right\\}\right]\\\
=&\mathbb{E}\big{[}\langle\Phi(x)|V^{\dagger}W^{\dagger}U^{\dagger}M_{1}UWV|\Phi(x)\rangle\big{]}\\\
=&\mathbb{E}\Bigg{[}\Bigg{\|}\begin{bmatrix}\mbox{\normalsize
0}&\mbox{\normalsize 0}\\\ \mbox{\normalsize
0}&I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}u_{11}\cdot U_{2:n}&u_{12}\cdot
U_{2:n}\\\ u_{21}\cdot U_{2:n}&u_{22}\cdot
U_{2:n}\end{bmatrix}\begin{bmatrix}w_{11}\cdot W_{2:n}&w_{12}\cdot W_{2:n}\\\
w_{21}\cdot W_{2:n}&w_{22}\cdot W_{2:n}\end{bmatrix}\begin{bmatrix}v_{11}\cdot
V_{2:n}&v_{12}\cdot V_{2:n}\\\ v_{21}\cdot V_{2:n}&v_{22}\cdot
V_{2:n}\end{bmatrix}\cdot\begin{bmatrix}\Phi_{1}(x)\\\
\Phi_{2}(x)\end{bmatrix}\Bigg{\|}^{2}\Bigg{]}\\\
=&\mathbb{E}\bigg{[}\Big{\|}(u_{21}w_{11}v_{11}+u_{22}w_{21}v_{11}+u_{21}w_{12}v_{21}+u_{22}w_{22}v_{21})\cdot
U_{2:n}W_{2:n}V_{2:n}\Phi_{1}(x)\\\
&\qquad\qquad+(u_{21}w_{11}v_{12}+u_{22}w_{21}v_{12}+u_{21}w_{12}v_{22}+u_{22}w_{22}v_{22})\cdot
U_{2:n}W_{2:n}V_{2:n}\Phi_{2}(x)\Big{\|}^{2}\bigg{]}\\\
=&\mathbb{E}\bigg{[}\Big{\|}(u_{21}w_{11}v_{11}+u_{22}w_{21}v_{11}+u_{21}w_{12}v_{21}+u_{22}w_{22}v_{21})\Phi_{1}(x)\\\
&\qquad\qquad+(u_{21}w_{11}v_{12}+u_{22}w_{21}v_{12}+u_{21}w_{12}v_{22}+u_{22}w_{22}v_{22})\Phi_{2}(x)\Big{\|}^{2}\bigg{]},\end{split}$

where the last equality holds since $U_{2:n}W_{2:n}V_{2:n}$ is unitary.
Therefore, $\tilde{m}_{\theta}(x)$ does not depend on $U_{2:n}$, $W_{2:n}$ and
$V_{2:n}$ and thus the second claim is proved.

##  Appendix B Proof of Theorem 2

We first introduce a lemma.

######  Lemma 1.

Assume noise model (1), (2), and, (3), and suppose $W$ has the form of

$W=W_{1}\otimes\cdots\otimes W_{n}.$

Define

$m^{jl}(x)=\mathbb{P}_{UWV\Phi(x)}\left\\{m^{1}=1\,|\,C=j,C^{\prime}=l\right\\}-\frac{1}{2}$

for $j\in\left\\{0,1,2,3\right\\}$ and $l\in\left\\{0,1,2,3\right\\}$. Then
the following statements hold.

  1. (a)

$m^{0l}(x)+m^{1l}(x)+m^{2l}(x)+m^{3l}(x)=0$ for all $l\in\\{0,1,2,3\\}$.

  2. (b)

$m^{00}(x)=m^{03}(x)=m(x)$, and $m^{01}(x)=m^{02}(x)$ with

$|m^{01}(x)-\cos\mu\cdot e^{-\frac{1}{2}\tau^{2}}\cdot
m(x)|\leq\frac{1}{2}|\sin\mu|\cdot e^{-\frac{1}{2}\tau^{2}}.$

Applying (a) of Lemma 1, we have

$\displaystyle\mathbb{P}\\{{m^{1}=1}\ |\ {C^{\prime}=l}\\}$
$\displaystyle=\mathbb{E}\left[{\mathbb{P}\\{{m^{1}=1}\ |\
{C,C^{\prime}=l}\\}}\ \middle|\ {C^{\prime}=l}\right]$
$\displaystyle=(1-3p)\cdot m^{0l}(x)+p\cdot m^{1l}(x)+p\cdot m^{1l}(x)+p\cdot
m^{1l}(x)$ $\displaystyle=(1-4p)\cdot
m^{0l}(x)+p\cdot\left(m^{0l}(x)+m^{1l}(x)+m^{2l}(x)+m^{3l}(x)\right)$
$\displaystyle=(1-4p)\cdot m^{0l}(x).$

It follows that

$\displaystyle\tilde{m}(x)$
$\displaystyle=\mathbb{E}\left[\mathbb{P}\\{{m^{1}=1}\ |\
{C^{\prime}=l}\\}\right]$ $\displaystyle=(1-3q)\cdot(1-4p)\cdot
m^{00}(x)+q\cdot(1-4p)\cdot m^{01}(x)+q\cdot(1-4p)\cdot
m^{02}(x)+q\cdot(1-4p)\cdot m^{03}(x)$
$\displaystyle=(1-4p)\cdot\left[(1-2q)\cdot m(x)+2q\cdot m^{01}(x)\right]$
$\displaystyle=(1-4p)\cdot\left[(1-2q)\cdot
m(x)+2q\cdot\left(u(x)+\cos\mu\cdot e^{-\frac{1}{2}\tau^{2}}\cdot
m(x)\right)\right]$ $\displaystyle\;\;\;\;\,(\text{where
}u(x)=m^{01}(x)-\cos\mu\cdot e^{-\frac{1}{2}\tau^{2}}\cdot m(x))$
$\displaystyle=(1-4p)\cdot\left(1-2q+\cos\mu\cdot
e^{-\frac{1}{2}\tau^{2}}\right)\cdot m(x)+(1-4p)\cdot 2q\cdot u(x).$

The desired inequality follows from $|u(x)|\leq\frac{1}{2}|\sin\mu|\cdot
e^{-\frac{1}{2}\tau^{2}}$ which holds by (b) of Lemma 1.

##  Appendix C Proof of Theorem 3

We apply the following lemma:

######  Lemma 2.

Under Assumption 2, the following inequality holds for any
$a,b,c,d\in\mathbb{R}$ with $a+b+c+d=0$.

$\frac{\ell(a)+\ell(b)+\ell(c)+\ell(d)}{4}\geq\frac{\ell(\frac{1}{3}|a|)+\ell(-\frac{1}{3}|a|)}{2}.$

The proof is given in the next section. Now we prove Theorem 3. The arguments
apply the idea of [3]. We first compute

$\displaystyle\mathbb{E}\Big{[}\tilde{R}_{N}(\theta)\,\Big{|}\,\psi_{1:N},Y_{1:N}\Big{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\mathbb{E}\Big{[}\tilde{R}_{N}(\theta)\,\Big{|}\,\psi_{1:N},Y_{1:N},C\Big{]}\Big{]}$
$\displaystyle=$
$\displaystyle(1-3p)\cdot\frac{1}{N}\sum_{i=1}^{N}\ell(m_{\theta}^{0}(\psi_{i})\cdot
Y_{i})+p\cdot\frac{1}{N}\sum_{i=1}^{N}\ell(m_{\theta}^{1}(\psi_{i})\cdot
Y_{i})+p\cdot\frac{1}{N}\sum_{i=1}^{N}\ell(m_{\theta}^{2}(\psi_{i})\cdot
Y_{i})+p\cdot\frac{1}{N}\sum_{i=1}^{N}\ell(m_{\theta}^{3}(\psi_{i})\cdot
Y_{i})$ $\displaystyle=$
$\displaystyle(1-4p)\hat{R}_{N}(\theta)+4p\cdot\frac{1}{N}\sum_{i=1}^{N}\frac{\ell(m_{\theta}(\psi_{i})\cdot
Y_{i})+\ell(m_{\theta}^{1}(\psi_{i})\cdot
Y_{i})+\ell(m_{\theta}^{2}(\psi_{i})\cdot
Y_{i})+\ell(m_{\theta}^{3}(\psi_{i})\cdot Y_{i})}{4}$

where

$m_{\theta}^{j}(x)=\langle\Phi(x)|(\sigma_{j}\otimes
I_{2^{n-1}})W(\theta)^{\dagger}M_{1}W(\theta)(\sigma_{j}\otimes
I_{2^{n-1}})|\Phi(x)\rangle-\frac{1}{2}$

Applying Lemma 1 with $U=I_{2^{n}}$, we have

$m_{\theta}(x)+m_{\theta}^{1}(x)+m_{\theta}^{2}(x)+m_{\theta}^{3}(x)=0$

Therefore, by lemma 2, it holds that

$\frac{1}{N}\sum_{i=1}^{N}\frac{\ell(m_{\theta}(\psi_{i})\cdot
Y_{i})+\ell(m_{\theta}^{1}(\psi_{i})\cdot
Y_{i})+\ell(m_{\theta}^{2}(\psi_{i})\cdot
Y_{i})+\ell(m_{\theta}^{3}(\psi_{i})\cdot Y_{i})}{4}\\\
\geq\frac{1}{N}\sum_{i=1}^{N}\bigg{(}\frac{\ell\big{(}\frac{1}{3}|m_{\theta}(\psi_{i})|\big{)}+\ell\big{(}-\frac{1}{3}|m_{\theta}(\psi_{i})|\big{)}}{2}\bigg{)},$

and this proves the claim.

##  Appendix D Proof of lemmas

###  D.1 Proof of Lemma 1

By Theorem 1, we may assume $U=U_{1}\otimes I_{2^{n-1}}$ and $V=V_{1}\otimes
I_{2^{n-1}}$. Fix any $l$, and compute

$\begin{split}m^{jl}(x)&=\mathbb{P}\\{{m^{1}=1}\ |\
{C=j,C^{\prime}=l}\\}-\frac{1}{2}\\\
&=\mathbb{E}\big{[}\mathbb{P}\\{{m^{1}=1}\ |\
{C=j,C^{\prime}=l,\epsilon}\\}\big{]}-\frac{1}{2}\\\
&=\mathbb{E}\left[\langle\Phi(x)|(\sigma_{j}\otimes
I_{2^{n-1}})^{\dagger}W^{\dagger}(U_{1}\otimes
I_{2^{n-1}})^{\dagger}M_{1}(U_{1}\otimes I_{2^{n-1}})W(\sigma_{j}\otimes
I_{2^{n-1}})|\Phi(x)\rangle\right]-\frac{1}{2}.\end{split}$

Next, we write

$\displaystyle U_{1}=\begin{bmatrix}u_{11}&u_{12}\\\
u_{21}&u_{22}\end{bmatrix},W_{1}=\begin{bmatrix}w_{11}&w_{12}\\\
w_{21}&w_{22}\end{bmatrix}$

and write $\Phi(x)$ in a block matrix form.

$\begin{split}\Phi(x)=\begin{bmatrix}\Phi_{1}(x)\\\
\Phi_{2}(x)\end{bmatrix},\quad\Phi_{1}(x),\Phi_{2}(x)\in\mathcal{C}^{2^{n-1}}.\end{split}$

Now if $j=0$, we have

$\begin{split}m^{0l}(x)&=\mathbb{E}\Bigg{[}\Bigg{\|}\begin{bmatrix}\mbox{\normalsize
0}&\mbox{\normalsize 0}\\\ \mbox{\normalsize
0}&I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}u_{11}I_{2^{n-1}}&u_{12}I_{2^{n-1}}\\\
u_{21}I_{2^{n-1}}&u_{22}I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}w_{11}W_{2:n}&w_{12}W_{2:n}\\\
w_{21}W_{2:n}&w_{22}W_{2:n}\end{bmatrix}\begin{bmatrix}\Phi_{1}(x)\\\
\Phi_{2}(x)\end{bmatrix}\Bigg{\|}^{2}\Bigg{]}-\frac{1}{2}\\\
&=\mathbb{E}\Big{[}\|{(u_{21}w_{11}+u_{22}w_{21})W_{2:n}\Phi_{1}(x)+(u_{21}w_{12}+u_{22}w_{22})W_{2:n}\Phi_{2}(x)}\|^{2}\Big{]}-\frac{1}{2}\\\
&=\mathbb{E}\Big{[}\|{(u_{21}w_{11}+u_{22}w_{21})\Phi_{1}(x)+(u_{21}w_{12}+u_{22}w_{22})\Phi_{2}(x)}\|^{2}\Big{]}-\frac{1}{2}.\end{split}$

If j=1, we have

$\begin{split}m^{1l}(x)&=\mathbb{E}\Bigg{[}\Bigg{\|}\begin{bmatrix}\mbox{\normalsize
0}&\mbox{\normalsize 0}\\\ \mbox{\normalsize
0}&I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}u_{11}I_{2^{n-1}}&u_{12}I_{2^{n-1}}\\\
u_{21}I_{2^{n-1}}&u_{22}I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}w_{11}W_{2:n}&w_{12}W_{2:n}\\\
w_{21}W_{2:n}&w_{22}W_{2:n}\end{bmatrix}\begin{bmatrix}\mbox{\normalsize
0}&I_{2^{n-1}}\\\ I_{2^{n-1}}&\mbox{\normalsize
0}\end{bmatrix}\begin{bmatrix}\Phi_{1}(x)\\\
\Phi_{2}(x)\end{bmatrix}\Bigg{\|}^{2}\Bigg{]}-\frac{1}{2}\\\
&=\mathbb{E}\Big{[}\|{(u_{21}w_{11}+u_{22}w_{21})W_{2:n}\Phi_{2}(x)+(u_{21}w_{12}+u_{22}w_{22})W_{2:n}\Phi_{1}(x)}\|^{2}\Big{]}-\frac{1}{2}\\\
&=\mathbb{E}\Big{[}\|{(u_{21}w_{11}+u_{22}w_{21})\Phi_{2}(x)+(u_{21}w_{12}+u_{22}w_{22})\Phi_{1}(x)}\|^{2}\Big{]}-\frac{1}{2}.\end{split}$

By similar procedure, we obtain

$m^{2l}(x)=\mathbb{E}\Big{[}\|{-(u_{21}w_{11}+u_{22}w_{21})\Phi_{2}(x)+(u_{21}w_{12}+u_{22}w_{22})\Phi_{1}(x)}\|^{2}\Big{]}-\frac{1}{2}$

and

$m^{3l}(x)=\mathbb{E}\Big{[}\|{(u_{21}w_{11}+u_{22}w_{21})\Phi_{1}(x)-(u_{21}w_{12}+u_{22}w_{22})\Phi_{2}(x)}\|^{2}\Big{]}-\frac{1}{2}.$

Now we define $a_{1}=u_{21}w_{11}+u_{22}w_{21}$ and
$a_{2}=u_{21}w_{12}+u_{22}w_{22}$, then we can write

$\displaystyle m^{0}(x)+m^{1}(x)+m^{2}(x)+m^{3}(x)$ $\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}\|{a_{1}\Phi_{1}(x)+a_{2}\Phi_{2}(x)}\|^{2}+\|{a_{1}\Phi_{2}(x)+a_{2}\Phi_{1}(x)}\|^{2}+\|{-a_{1}\Phi_{2}(x)+a_{1}\Phi_{1}(x)}\|^{2}+\|{a_{1}\Phi_{1}(x)-a_{2}\Phi_{2}(x)}\|^{2}\Big{]}-2$
$\displaystyle=$
$\displaystyle\mathbb{E}\Big{[}2(|a_{1}|^{2}+|a_{2}|^{2})(\|{\Phi_{1}(x)}\|^{2}+\|{\Phi_{2}(x)}\|^{2})\Big{]}-2$
$\displaystyle=$ $\displaystyle 2\cdot 1\cdot\|{\Phi(x)}\|^{2}-2$
$\displaystyle=$ $\displaystyle 0,$

where the third equality holds since the vector $(a_{1},a_{2})$ is the second
row of the matrix $U_{1}W_{1}$ which is unitary. Hence, (a) is proved.

Next, define

$\displaystyle\Psi(x)=\begin{bmatrix}\Psi_{1}(x)\\\
\Psi_{2}(x)\end{bmatrix}=W|\Phi(x)\rangle$

and compute

$\begin{split}m^{00}(x)&=\mathbb{E}\big{[}\mathbb{P}\\{{\text{measurement}=1}\
|\ {\epsilon,C=0,C^{\prime}=0}\\}\big{]}-\frac{1}{2}\\\
&=\mathbb{E}\big{[}\langle\Phi(x)|W^{\dagger}U^{\dagger}M_{1}UW|\Phi(x)\rangle\big{]}-\frac{1}{2}\\\
&=\mathbb{E}\Bigg{[}\Bigg{\|}\left(\cos\frac{\mu+\epsilon}{2}-i\sin\frac{\mu+\epsilon}{2}\right)\cdot\begin{bmatrix}\mbox{\normalsize
0}&\mbox{\normalsize 0}\\\ \mbox{\normalsize
0}&I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}\Psi_{1}(x)\\\
\Psi_{2}(x)\end{bmatrix}\Bigg{\|}^{2}\Bigg{]}-\frac{1}{2}\\\
&=\|{\Psi_{2}(x)}\|^{2}-\frac{1}{2}(=m(x)),\end{split}$

and

$\begin{split}m^{01}(x)&=\mathbb{E}\big{[}\mathbb{P}\\{{\text{measurement}=1}\
|\ {\epsilon,C=0,C^{\prime}=1}\\}\big{]}-\frac{1}{2}\\\
&=\mathbb{E}\Bigg{[}\Bigg{\|}\begin{bmatrix}\mbox{\normalsize
0}&\mbox{\normalsize 0}\\\ \mbox{\normalsize
0}&I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}\cos\frac{\mu_{1}+\epsilon_{1}}{2}\cdot
I_{2^{n-1}}&-i\sin\frac{\mu_{1}+\epsilon_{1}}{2}\cdot I_{2^{n-1}}\\\
-i\sin\frac{\mu_{1}+\epsilon_{1}}{2}\cdot
I_{2^{n-1}}&\cos\frac{\mu_{1}+\epsilon_{1}}{2}\cdot
I_{2^{n-1}}\end{bmatrix}\begin{bmatrix}\Psi_{1}(x)\\\
\Psi_{2}(x)\end{bmatrix}\Bigg{\|}^{2}\Bigg{]}-\frac{1}{2}\\\
&=\mathbb{E}\left[\Big{\|}{-i\sin\frac{\mu_{1}+\epsilon_{1}}{2}\cdot\Psi_{1}(x)+\cos\frac{\mu_{1}+\epsilon_{1}}{2}\cdot\Psi_{2}(x)}\Big{\|}^{2}\right]-\frac{1}{2}\\\
&=\mathbb{E}\left[\frac{1}{2}+\cos(\mu_{1}+\epsilon_{1})\cdot\left(\|{\Psi_{2}(x)}\|^{2}-\frac{1}{2}\right)-\sin(\mu_{1}+\epsilon_{1})\cdot\text{Re}(i\Psi_{1}(x)^{\dagger}\Psi_{2}(x))\right]-\frac{1}{2}\\\
&=\cos\mu_{1}\cdot m(x)\cdot e^{-\frac{1}{2}\tau^{2}}+\sin\mu_{1}\cdot
e^{-\frac{1}{2}\tau^{2}}\cdot\text{Im}(\Psi_{1}(x)^{\dagger}\Psi_{2}(x)).\end{split}$

By similar computations we have

$\displaystyle m^{03}(x)$
$\displaystyle=\|{\Psi_{2}(x)}\|^{2}-\frac{1}{2}=m^{00}(x)$ $\displaystyle
m^{02}(x)$ $\displaystyle=\cos\mu_{1}\cdot m(x)\cdot
e^{-\frac{1}{2}\tau^{2}}+\sin\mu_{1}\cdot
e^{-\frac{1}{2}\tau^{2}}\cdot\text{Im}(\Psi_{1}(x)^{\dagger}\Psi_{2}(x))=m^{01}(x).$

Note that

$|\text{Im}(\Psi_{1}(x)^{\dagger}\Psi_{2}(x))|\leq|\Psi_{1}(x)^{\dagger}\Psi_{2}(x)|\leq\frac{|\Psi_{1}(x)|^{2}+|\Psi_{2}(x)|^{2}}{2}=\frac{1}{2}.$

This proves (b).

###  D.2 Proof of Lemma 2

Let us define

$s=\frac{1}{2}\cdot\max\left\\{|a+b|,|a+c|,|a+d|\right\\}.$

Then we have

$\frac{\ell(a)+\ell(b)+\ell(c)+\ell(d)}{4}\geq\frac{\ell(s)+\ell(-s)}{2},$

since

$\displaystyle\frac{\ell(a)+\ell(b)+\ell(c)+\ell(d)}{4}$
$\displaystyle\geq\frac{1}{2}\cdot\left[\ell\left(\frac{a+b}{2}\right)+\ell\left(\frac{c+d}{2}\right)\right]\textnormal{\quad
by Jensen's inequality}$
$\displaystyle=\frac{1}{2}\cdot\left[\ell\left(\frac{a+b}{2}\right)+\ell\left(-\frac{a+b}{2}\right)\right]\textnormal{\quad
since $a+b+c+d=0$}$
$\displaystyle=\frac{1}{2}\cdot\left[\ell\left(\frac{|a+b|}{2}\right)+\ell\left(-\frac{|a+b|}{2}\right)\right]$

and the above procedure also holds when $(a+b,c+d)$ is replace by $(a+c,b+d)$
or $(a+d,b+c)$. Note that $t\mapsto\frac{\ell(t)+\ell(-t)}{2}$ is
nondecreasing at $t>0$ because of the convexity of $\ell$. Now by definition
of $s$, we have

$|a|=|b+c+d|=\frac{1}{2}|(b+c)+(c+d)+(d+c)|=\frac{1}{2}|(a+b)+(a+c)+(a+d)|\leq
3s,$

and therefore

$\frac{\ell(s)+\ell(-s)}{2}\geq\frac{\ell(\frac{1}{3}|a|)+\ell(-\frac{1}{3}|a|)}{2}.$

The desired inequality immediately follows from these observations.


# Live 3D Portrait: Real-Time Radiance Fields for
Single-Image Portrait View Synthesis

Alex Trevithick This project was initiated and substantially carried out
during an internship at NVIDIA. University of California, San Diego  Matthew
Chan  NVIDIA  Michael Stengel  NVIDIA  Eric Ryan Chan11footnotemark: 1
Stanford University  Chao Liu  NVIDIA  Zhiding Yu  NVIDIA
Sameh Khamis  NVIDIA  Manmohan Chandraker  University of California, San Diego
Ravi Ramamoorthi  University of California, San Diego  Koki Nagano  NVIDIA

###### Abstract

We present a one-shot method to infer and render a photorealistic 3D
representation from a single unposed image (e.g., face portrait) in real-time.
Given a single RGB input, our image encoder directly predicts a canonical
triplane representation of a neural radiance field for 3D-aware novel view
synthesis via volume rendering. Our method is fast (24 fps) on consumer
hardware, and produces higher quality results than strong GAN-inversion
baselines that require test-time optimization. To train our triplane encoder
pipeline, we use only synthetic data, showing how to distill the knowledge
from a pretrained 3D GAN into a feedforward encoder. Technical contributions
include a Vision Transformer-based triplane encoder, a camera data
augmentation strategy, and a well-designed loss function for synthetic data
training. We benchmark against the state-of-the-art methods, demonstrating
significant improvements in robustness and image quality in challenging real-
world settings. We showcase our results on portraits of faces (FFHQ) and cats
(AFHQ), but our algorithm can also be applied in the future to other
categories with a 3D-aware image generator.

Figure 1:  Given a single RGB input image, our method generates 3D-aware
images and geometry of an object (e.g., faces [top row] and cats [bottom row,
left]) in real-time, while the state-of-the-art 3D GAN inversion [eg3d2022]
does not generate a satisfactory result after 20 mins of generator fine-tuning
[roich2021pivotal] (top right). Our method can also be applied to a video in a
frame-by-frame fashion for video-based novel view synthesis (bottom row,
right). Ours (LT) refers to a lightweight faster version of our model that has
almost the same quality as the full model.

Figure 2: Comparison to the state-of-the-art 3D GAN[eg3d2022] with test-time
fine tuning[roich2021pivotal] (EG3D-PTI). Single-view 3D GAN inversion
approaches trade off the 2D reconstruction quality and the 3D effects. When
fine tuned longer (7500 iterations), EG3D-PTI can capture the same fine-scale
details as ours ($LPIPS=0.199$), but the quality of another view starts to
degrade. On the other hand, our method captures out-of-domain details (e.g.,
emblem) in one-shot while producing realistic rendering of another view, and
operating in real-time.

##  1 Introduction

Digitally reproducing the 3D appearance of an object from a single image is a
long-standing goal for computer graphics and vision. Interactive synthesis of
photorealistic novel views opens new possibilities for AR/VR, and for 3D
telepresence and videoconferencing when applied to humans. In this work, we
propose a technique to infer a 3D representation for real-time view synthesis
given a single portrait-style input image (e.g., of a human face, see Fig. 1).

Recently, 3D aware-image generation approaches (e.g., [eg3d2022, deng2022gram,
epigraf]) demonstrated unconditional generation of photorealistic 3D
representations from a collection of single-view 2D images by combining NeRF-
based representations [mildenhall2020nerf] and GANs
[goodfellow2014generative]. Notably, EG3D [eg3d2022] proposed an efficient
triplane 3D representation and demonstrated real-time 3D-aware image rendering
with quality comparable to 2D GANs. Once trained, the 3D GAN generators can be
frozen and used for single-image 3D reconstruction tasks via GAN inversion
[Karras2020stylegan2] and test-time fine tuning [roich2021pivotal]. However,
there are a few challenges in this 3D-GAN inversion-based methods. (1) Due to
the multi-view nature of training a NeRF, it needs careful optimization
objectives and additional 3D priors [yin2022spi, xie2022high] in the single
view setting to avoid unsatisfactory results on novel views and corrupted
geometry (see Fig. 7). Fig. 2 shows the tradeoff in the SOTA single-view 3D
GAN inversion pipeline. (2) The test-time optimization requires an accurate
camera pose as input or to be jointly optimized [ko20233d3dganinversion]. (3)
The above optimization for every single image is time-consuming, limiting the
technique for real-time video applications.

In this paper, we present a one-shot approach to lift an input 2D portrait
image to 3D in real-time (24fps on consumer hardware, see Tab. 1). Unlike
previous work that reuses a pre-trained generator, we train an encoder end-to-
end that directly predicts the triplane 3D features from a single input image.
In contrast to prior works that use multiview real image acquisition setups,
we do not need any real images at all, nor do we require time-consuming
physically-based rendering of high-quality and expensive face assets.

Instead, we fully supervise the training of our triplane encoder for novel
view synthesis using multiview-consistent synthetic data generated from a pre-
trained 3D GAN. Together with our data augmentation strategies and
Transformer-based encoder, we present a model which can handle challenging
real-world input images including occlusion and three-quarter views. We
showcase our results on human and cat face categories in this paper, but the
methodology can apply to any category for which 3D-aware image generators are
available. Our work may motivate applications such as temporally consistent
view synthesis; Fig. 1 (bottom right) shows our method applied to a video in a
frame-by-frame fashion without any special handling.

In summary, contributions of our work include:

  * •

We propose a feed forward encoder model to directly infer a triplane 3D
representation from an input image. No test-time optimization is needed.

  * •

We present a new strategy for training a feed forward triplane encoder for 3D
inversion using only synthetic data generated from a pre-trained 3D-aware
image generator.

  * •

We demonstrate that our method can infer a photorealistic 3D representation in
real-time given a single unposed image. Together with our Transformer-based
encoder and on-the-fly augmentation strategy, our method can robustly handle
challenging input images of side views and occlusions.

Time | H.NeRF | ROME | EG3D-PTI | Ours | Ours (LT)
---|---|---|---|---|---
Encoding | 60s |  55ms† | 2 mins | 40ms | 16ms
Render | 58ms | 27ms | 24ms | 24ms | 24ms
Table 1: Time taken to lift the input image to 3D (Encoding) and render
(Render) a 3D representation given an input image on a single RTX 3090 GPU.
The end-to-end runtime with our model and our lightweight model (LT) is
significantly faster than NeRF-based baselines. †ROME employs 2D-based neural
rendering with mesh-based neural textures, producing the output at 256x256
resolution; it also requires a segmentation mask and detected keypoints from
off-the-shelf models which requires around 200ms.

##  2 Related work

Our work touches on light fields, few-shot view synthesis, learning with
synthetic data, 3D-aware portrait generation, and GAN inversion. Our focus is
on real-time view synthesis from a single image, and we do not address
portrait relighting or editing. Tab. 1 summarizes runtime for inferring 3D
representations from an input and rendering. Our one-shot method is three
orders of magnitude faster than the NeRF-based state-of-the-art methods for
inference, enabling a real-time pipeline.

#### Light Fields and Image-Based Rendering.

View synthesis or image-based rendering has a long history in computer
graphics and vision [ChenWilliams, McMillan], and has often been framed in
terms of reconstructing the light field [Levoy, Gortler]. However, those
methods typically required hundreds of views. Subsequent light field
approaches demonstrated few-shot general [Nima] and even single image view
synthesis for categories [PratulICCV], but required light field camera
training data. More recently, neural-field based approaches
[mildenhall2020nerf, Yiheng2022neuralfield] combine recent neural implicit 3D
representations [park2019deepsdf, mescheder2019occupancy, chen2019learning,
sitzmann2019srns] with volume rendering for novel view synthesis, but require
a large number of input images per scene.

#### Few-shot novel view synthesis.

Some recent work extends NeRF for training from even a single view
[Xu_2022_SinNeRF] or for few-shot novel view synthesis using fully implicit 3D
representations [li2022symmnerf, yu2021pixelnerf, grf2020, jang2021codenerf],
3D convolutions [mvsnerf, xianggang2022pvserf], or Transformers
[lin2023visionnerf, wang2021ibrnet]. However these approaches do not generate
novel views in real-time. Moreover, all of the above approaches need multi-
view images to train their models. Our method, on the other hand, only needs
synthetic images generated from a pre-trained 3D GAN, which is trained by a
collection of single-view images. FWD [Cao2022FWD] builds on top of
SynSin[wiles2020synsin] for real-time novel view synthesis using depth-based
image warping, but requires depth data from multiview stereo or a depth
sensor. Yet another family of approaches is the geometry-free
method[sajjadi2022scene, rombach2021geometryfree, ren2022look], but they need
a large number of images to learn precise ray transformations; otherwise it
may lead to blurry or multiview inconsistent results.

#### Learning with synthetic data.

Synthetic data provides useful supervision for training a deep learning model
when ground truth data is not available. Previous methods used synthetic data
for various deep learning-based tasks such as dense visual alignment
[peebles2022gansupervised], 3D face reconstruction [pan2020gan2shape,
wood2022dense] and analysis [Wood_2021_fakeit], portrait normalization
[zhang2020portrait, nagano2019deepfacenorm], and semantic segmentation
[zhang21datasetgan, Tritrong2021RepurposeGANs]. Some previous work used
synthetic face portrait images generated by rendering 3D face assets using a
phycally-based pathtracer to train a model for portrait relighting
[yeh2022learning] or relighting and view synthesis [sun2021nelf]. Since the CG
rendering exhibits a synthetic look, they need an additional step to adapt to
real images. Other concurrent work [ko20233d3dganinversion] uses a discrete
number of pre-generated synthetic images from a 3D-aware generator [eg3d2022]
for 3D GAN inversion tasks. Instead, we generate an unlimited amount of
synthetic data in the training loop and show that on-the-fly camera
augmentation is critical for generalization to real images for synthetic data
training.

#### 3D-aware portrait generation and manipulation.

For a well-known category of object, such as human faces, previous work
[Khakhulin2022ROME, Gao-portraitnerf, athar2022rignerf, hong2021headnerf,
Mihajlovic:ECCV2022, wang2022morf, abs-1802-05384, nagano2018pagan,
kim2018deep] used 3D face priors for few-shot portrait synthesis. While the
face priors provide additional capabilities for facial manipulations and
expression retargeting [seol2011retargeting], they don’t generalize beyond
humans. Recently, 3D aware-image generation approaches [hologan, graf,
chan2020pi, Niemeyer2020GIRAFFE] started to tackle the problem of
unconditional generation of photorealistic 3D representations from a
collection of single-view 2D images. By combining neural volumetric
rendering[mildenhall2020nerf] and generative adversarial networks
(GANs)[goodfellow2014generative], recent 3D GAN approaches[eg3d2022,
orel2022stylesdf, gu2021stylenerf, deng2022gram, xiang2022gramhd, epigraf,
Zhou2021CIPS3D, zhang2022mvcgan, xu2021volumegan, rebain2022lolnerf] started
to demonstrate an ability to generate high-resolution multi-view consistent
images and geometry of a category of objects. We adapt the efficient triplane
3D representation from EG3D [eg3d2022] and demonstrate single-view novel view
synthesis on similar categories.

#### 3D GAN inversion.

Following the success of GAN inversion in 2D domains for image editing and
manipulations[richardson2021encoding, tov2021e4e, dinh2021hyperinverter,
alaluf2021restyle, wang2021HFGI], existing 3D GAN inversion methods
[ko20233d3dganinversion, sun2022ide, lin20223dganinversion] project a given
image to variants of the pre-trained StyleGAN2 latent space
[Karras2020stylegan2, abdal2019image2stylegan]. Assuming multiview images,
FreeStyleGan [leimkuhler] proposes to map projected camera coordinates to a
subject-specific StyleGAN2 latent space which allows the subject to be
rendered from specified cameras under the constraints of the StyleGAN prior.
While this global latent space provides an additional ability for 3D-aware
portrait editing, the StyleGAN2 latent space trades off reconstruction
fidelity for editability, making the exact reconstruction of the input image
challenging. Thus, existing 3D GAN inversion approaches require an approximate
camera pose and slight generator weight tuning [roich2021pivotal,
Feng2022nearperfect] at test time to reconstruct out-of-domain input images.
Our feed forward encoder takes an unposed image as input and does not need
test-time optimization for camera poses unlike concurrent work
[ko20233d3dganinversion].

#### Talking-head generators.

Given a single target portrait and a driving video, recent talking-head
generators can reenact the portrait by transferring facial expressions and
head poses from the driver video [wang2022latent, hong2022depth, Zakharov20,
Drobyshev22MP, wang2021facevid2vid, tps2022, doukas2020headgan]. Trained by
video datasets, their methods mainly focuse on talking-head video generation
by manipulating avatar poses and expressions within a 2D portrait. As such,
they do not predict volumetric representations that allow free viewpoint
rendering including background and do not provide dense 3D geometry like our
method. Therefore, we do not compare to these approaches.

Figure 3: Inference and training outline for our pipeline. At inference, we
take an unposed image, and extract low resolution features
$\bm{F}_{\text{low}}$ with a DeepLabv3 backbone. These features are fed to a
ViT yielding $\bm{F}$ and then concatenated after convolution with high-
resolution features $\bm{F}_{\text{high}}$ before being decoded with a ViT and
convolutions to a triplane representation. These features condition the
volumetric rendering process which yields depth, feature, color, and
superresolved images. During training, we sample an identity from EG3D and
then render two supervision views. The first serves as the input to our
encoder, which predicts a triplane, which conditions volume rendering from the
same two views. The rendering results are compared with those of EG3D as
outlined in Sec. 4. Features are visualized for illustration.

##  3 Preliminaries: Triplane-based 3D GAN

We first give an overview of the state-of-the-art 3D GAN method, EG3D,
[eg3d2022] from which our method will distill knowledge. EG3D learns
unconditional 3D-aware image generation from a collection of single-view
images and corresponding noisy camera poses, where each image has resolution
$512\times 512$. As mentioned in Sec. 2, EG3D makes use of a hybrid triplane
representation to condition the neural volumetric rendering process, whereby
three 2D feature grids are stored along each of the three canonical
planes–$xy$, $xz$, $yz$. Using a StyleGAN2 generator [Karras2020stylegan2],
EG3D maps a noise vector and conditioning camera poses to a triplane
representation $\bm{T}\in\mathbb{R}^{256\times 256\times 96}$ which
corresponds to the $3$ axis-aligned planes, each with $32$ channels. These
features condition the neural volumetric rendering.

To assign a point $\bm{x}\in\mathbb{R}^{3}$ with its feature, color and volume
density, $(\mathbf{f},\mathbf{c},\sigma)$, a lightweight MLP decodes the three
feature vectors gathered by projecting $\bm{x}$ to each of the canonical
planes:

$(\mathbf{f},\mathbf{c},\sigma)=\text{MLP}(\Phi(\bm{f}_{xy},\bm{f}_{xz},\bm{f}_{yz})),$
(1)

where $\bm{f}_{ij}$ are the features gathered by projecting $\bm{x}$ to the
$ij$ plane and bilinearly interpolating the nearby features, and $\Phi$ is the
mean operator. Note that output values including the color are independent of
viewing direction and only depend on $\bm{x}$. By accumulating many points
along rays, and performing volume rendering [Max:1995] as in NeRF
[mildenhall2020nerf], one may render a feature image
$\bm{I}_{f}\in\mathbb{R}^{32\times 128\times 128}$ and a raw neural rendering
RGB image $\bm{I}_{128}\in\mathbb{R}^{3\times 128\times 128}$ from a given
camera pose. In practice, $\bm{I}_{128}$ corresponds to the first three
channels of the feature image $\bm{I}_{f}$.

We additionally extract a dense depth map $\bm{I}_{D}\in\mathbb{R}^{128\times
128}$ from this volume rendering, which we use later to supervise our model.
The neural rendered images $\bm{I}_{128}$ and $\bm{I}_{f}$ are then fed to a
2D superresolution network, which yields the final superresolved rendering
output:

$\text{SuperRes}(\bm{I}_{f},\bm{I}_{128})=\bm{I}_{512}\in\mathbb{R}^{3\times
512\times 512}.$ (2)

This 3D GAN pipeline is trained end-to-end following 2D GAN training with a 2D
(dual) discriminator. The reader is referred to the original paper [eg3d2022]
for full details.

The efficient design of EG3D allows rendering from a triplane at 42 fps on the
RTX 3090. At the same time, EG3D provides comparable quality to even the
state-of-the-art 2D GANs by FID. These attributes provide a strong basis for
supervising our encoder-based method using EG3D-generated synthetic data.

Figure 4: Qualitative comparisons on one FFHQ subject between HeadNeRF, ROME
(256$\times$ 256 resolution), EG3D-PTI, and ours. HeadNeRF and ROME require
face and background segmentation,do not synthesize the full portrait and fail
to capture detailed 3D geometry. EG3D-PTI reconstructs a high-quality 2D image
after 20 mins of tuning, but also produces a degenerated 3D shape. Our 40ms
prediction recovers both high-fidelity 3D geometry as well as 2D image quality
similar to EG3D-PTI.

##  4 Method

Figure 5: FFHQ and AFHQ qualitative results from our model (left) and our
lightweight model (LT) (right). We showcase reconstructed input and novel
views, or the learned geometry. In the bottom-right, note our model’s ability
to infer structure consistent with the input behind occlusion. Credits to
YuChen Cheng, Montclair Film, Lydia Liu.

Our goal is to distill the knowledge of a fully trained EG3D generative model
(learned over a category or set of categories) into a feedforward encoder
pipeline that can directly map an unposed image to a canonical triplane 3D
representation111 Note that each category has a different notion of canonical
representation: for human faces, the center of the head is the origin, and
planes orthogonally intersect the head up-to-down, left-to-right, and front-
to-back.  which can be decoded with a NeRF. This pipeline requires only a
single feedforward network pass, thus avoiding the expensive GAN inversion
process, while allowing free viewpoint re-rendering of the input in real-time.
Note that our contribution focuses on the image-to-triplane encoder and
associated synthetic training method, as shown in the pipeline of Fig. 3. We
make use of the MLP volume renderer and superresolution architectures from
EG3D as per Eqns 1 and 2 and train all the components end-to-end. In Tab. 1,
the top row shows that our image to triplane inference runs at up to 60 fps
(16 ms), while rendering has identical performance to EG3D (bottom row of Tab.
1).

###  4.1 Triplane encoder

We note that inferring a canonicalized 3D reprentation (i.e., the inferred 3D
representation is frontalized and aligned) from an arbitrary RGB image while
simultaneously synthesizing precise subject-specific details from the input is
a highly non-trivial task. We break this challenge into the two-fold goals: 1)
to create a canonicalized 3D representation of the subject from an image, and
2) to render high-frequency person-specific details. We note these goals are
often at odds with one another, and exemplify the bias-variance tradeoff
whereby the output will resemble the input well, but may not be correctly
canonicalized in 3D (see Fig. 13), or the output will have the correct 3D
structure, but not resemble the 2D input image (see Fig. 12). Our encoder
manages to accomplish both of these goals simultaneously. Specifically, we
develop and train a hybrid convolutional-Transformer encoder, $\mathbf{E}$,
which maps from an unposed RGB image, $\bm{I}$, to the _canonical_ triplane
representation.

As seen in the upper half of Fig. 3, the architecture of our encoder begins
with a fast convolutional backbone, DeepLabV3 [chen2017rethinking], which
extracts robust low-resolution features,
$\bm{F}_{\text{low}}=\text{DeepLabV3}(\bm{I})$. These features are then fed to
a Vision Transformer (and CNN) which gives a global inductive bias to the
intermediate output features,

$\bm{F}=\mathrm{Conv}(\mathrm{ViT}(\bm{F}_{\text{low}})),$ (3)

where $\mathrm{Conv}$ is a CNN and $\mathrm{ViT}$ is the Vision Transformer
Block from Segformer [xie2021segformer] with efficient self-attention. We
choose the Segformer ViT for two reasons: 1) it was designed to quickly map to
a high-resolution output space similar to a triplane, and 2) the efficient
self-attention mechanism allows the use of high-resolution intermediate
feature maps so that all information flows from input to triplane.

We consider the ViT features as having successfully created a canonicalized 3D
representation of the subject (completing the step 1 above), and found during
our experimentation that this shallow encoder is sufficient to reasonably
canonicalize a subject, yet cannot represent important high-frequency or
subject-specific details like strands of hair or birthmarks.

In order to simultaneously complete the second step (adding high-frequency
detail), we next reincorporate high-resolution image features. We
convolutionally encode the image again with only a single downsampling stage
with encoder $\mathbf{E}_{\text{high}}$ to obtain features
$\bm{F}_{\text{high}}=\mathbf{E}_{\text{high}}(\bm{I})$. These are
concatenated with the extracted global features and passed through another
Vision Transformer, which is finally decoded to a triplane with convolutions
as seen in Fig. 3. Thus, the output of our encoder has the following form:

$\bm{T}=\mathbf{E}(\bm{I})=\mathrm{Conv}(\mathrm{ViT}(\bm{F}\oplus\bm{F}_{\text{high}})),$
(4)

where $\oplus$ denotes concatenation along the channel axis, and $\bm{T}$ is
triplane feature representation used in Sec. 3.

###  4.2 Training with synthetic data

As seen in Fig. 3 in the training step, we train our triplane encoder with
synthetic data. Sampling a latent vector and passing it through the EG3D
generator yields a corresponding triplane, $\bm{T}$. Given camera parameters
$\bm{P}$ (a focal length, principal point, camera orientation and position),
we can render any image from the frozen EG3D generator and $\bm{T}$. At each
gradient step, we synthesize two images of the same identity (same latent
code) from a reference (input) camera $\bm{P}_{\mathrm{ref}}$ and another
camera $\bm{P}_{\mathrm{mv}}$ for multiview supervision. Using the same
notation as in Sec. 3, each rendering pass will give us four images:
$\bm{I}_{f}$, $\bm{I}_{128}$, $\bm{I}_{512}$, and $\bm{I}_{D}$ as seen in Fig.
3.

Again as shown in Fig. 3, the input to our encoder is the high-resolution
image $\bm{I}_{512}$ (highlighted in red) rendered from the input camera
$\bm{P}_{\mathrm{ref}}$, so that $\overline{\bm{T}}=\mathbf{E}(\bm{I}_{512})$.
We then use $\overline{\bm{T}}$ to condition the volume rendering process from
both camera $\bm{P}_{\mathrm{ref}}$ and $\bm{P}_{\mathrm{mv}}$, to get two
more sets of four images, which we denote as $\overline{\bm{I}}_{f}$,
$\overline{\bm{I}}_{128}$, $\overline{\bm{I}}_{512}$, and
$\overline{\bm{I}}_{D}$. Our loss intuitively compares those quantities
synthesized by EG3D and those created by our encoder, along with a generative
adversarial objective as follows:

$\displaystyle L=\
L_{\text{tri}}+L_{\text{col}}+L_{\text{LPIPS}}+L_{\text{feat}}+\lambda_{1}L_{\text{adv}}+\lambda_{2}L_{\text{cate}}$
(5)

$L_{\text{tri}}$ is the L1 loss between $\bm{T}$ and $\overline{\bm{T}}$;
$L_{\text{col}}$ is the mean L1 loss computed between both sets of pairs
$(\bm{I}_{128},\overline{\bm{I}}_{128})$ and
$(\bm{I}_{512},\overline{\bm{I}}_{512})$; $L_{\text{LPIPS}}$ is the LPIPS
perceptual loss [zhang2018perceptual] computed over both sets of pairs
$(\bm{I}_{128},\overline{\bm{I}}_{128})$ and
$(\bm{I}_{512},\overline{\bm{I}}_{512})$; $L_{\text{feat}}$ is the mean L1
loss computed between the pairs $(\bm{I}_{f},\overline{\bm{I}}_{f})$;
$L_{\text{adv}}$ is the adversarial loss using a pretrained dual discriminator
from EG3D which is fine-tuned during training; $\lambda_{1}$ is $0.1$ for the
reference image or $0.025$ for the multiview image; and $L_{\text{cate}}$ is
an optional category-specific loss. For human faces, we use $\lambda_{2}$ to
be $1$ with face identity features from ArcFace [deng2018arcface] following
practice in 2D GAN inversion [richardson2021encoding, tov2021e4e]. For cat
faces, we set $\lambda_{2}$ to $0$. This objective is optimized end-to-end,
i.e., with respect to all of the parameters of the encoder, rendering and
upsampling modules. Note that the rendering, upsampling, and dual
discriminator modules are all fine-tuned from the pretrained EG3D. However,
the dual discriminator in our pipeline doesn’t rely on _any_ real data;
instead, we train this discriminator to differentiate between images rendered
from our encoder model and images rendered from the frozen EG3D. An ablation
showing its effectiveness is provided in Tab. 5 and Fig. 14.

#### On-the-fly augmentation.

Naively optimizing this objective will yield a model which performs almost
perfectly on synthetic data, but lacks the ability to generalize to real
images (see Fig. 13). In order to remedy this, we augment the standard EG3D
rendering method which assumes a fixed camera roll, focal length, principal
point and distance from subject. In contrast, we sample all four of these
values from random distributions to choose the camera parameters
$\bm{P}_{\mathrm{ref}}$. The details of these distributions for each dataset
are given in the supplement. For $\bm{P}_{\mathrm{mv}}$, we choose fixed
values as in the EG3D model. For $\bm{P}_{\mathrm{ref}}$, we sample the
cameras from a pitch range of $\pm 26^{\circ}$ and yaw range of $\pm
49^{\circ}$ relative to the front of a human face. For $\bm{P}_{\mathrm{mv}}$,
we sample the cameras from a pitch range of $\pm 26^{\circ}$ and yaw range of
$\pm 36^{\circ}$ relative to the front of a human face. This allows the
supervision of our model to happen with highly variable camera poses, forcing
the model to learn to effectively canonicalize and infer from challenging
images as seen in Fig. 5.

#### Implementation details

Before training with the full adversarial objectives in Eqn. 5, we warm up the
model by training over 30k iterations without the adversarial loss and
continue to train the model with the full loss functions in Eqn. 5 over 220k
iterations. Since we sample two camera poses per iteration (with batch size
32), we effectively use over 16 million images during the training, which is
not obtainable from real images (nor even physically-based rendered images) in
practice. For full implementation details, please refer to the supplement. We
train two encoders with two different compute budgets: ”Ours”, which has 87M
parameters and ”Ours (LT)”, a lightweight model (LT) which has 63M parameters.
The main difference between the two is in resolution of the intermediate
feature maps, which result in fewer parameters in the LT model, but both
contain the same structure outlined above. ”Ours” runs in 22ms on a single
A100 GPU (where rendering takes 15ms) and 40ms on RTX 3090 as seen in Table 1.
”Ours (LT)” runs in just 16ms on RTX 3090, while retaining strong performance
(see the numerical evaluations in Tabs. 2 and 3). Figures 1 and 5 show the
qualitative outputs from both models.

##  5 Results

Figure 6: Qualitative results displaying our model’s reconstruction on the
input view, and the learned geometry from the frontal view. The reconstructed
geometry remains faithful to the input image. Credits to Devon Weller, Jamie,
SupportPDX, Mary Sawatzky, map, Herzliya Conference, Helse Midt-Norge, Tom
Munnecke, pter tr, UGA CAES/Extension, Rare Cancers Australia, Vladimir
Agafonkin, Michael E. Macmillan, Nguyen Hung Vu. Figure 7: Qualitative results
displaying our model in comparison to baseline methods HeadNeRF, ROME, and
EG3D-PTI, comparing the image quality (left) and reconstructed geometry
(right). EG3D-PTI occasionally exhibits corrupted 3D geometry (2nd and 4th
rows) when the input is side view, indicating that the learned 3D prior alone
is not enough to ensure robust reconstruction. Credit to U.S. Dept. of HUD,
Cristina Cifuentes, Rainforest Action Network, CENA MINEIRA. Figure 8: Ground
truth comparisons on the H3DS dataset including ground truth geometry (second
row) and unseen validation view (third row). Since the H3DS ground truth data
has inconsistent lighting, the lighting discrepancy is expected for the
validation view. Figure 9: Qualitative results displaying our model’s ability
to lift StyleGAN2-generated drawings and paintings to 3D. These results
display the generalizability of our model, as it can canonicalize out-of-
domain drawings and portraits, lifting them to 3D.  Figure 10: Comparison to
baselines at various input pitch angles. Credits to Bjørnar Tollaksen, Juliana
Martuscelli, The 621st Contingency Response Wing, U.S. Army Security
Assistance Command, Sam Wadman, Laity Lodge Family Camp, U.S. President’s
Malaria Initiative, SickKids Foundation.  Figure 11: Comparison to baselines
at various input yaw angles. Credit to Lionel AZRIA, justinkim1, nonorganical,
Paradox Wolf, Agência Senado, Ademir Brito, Ariana Vincent, Jay Weenig, John
Benson, Seong Bae.

We evaluate methods for single-view novel view synthesis on 3 main aspects (1)
2D image reconstruction (LPIPS [zhang2018perceptual], DISTS [ding2020iqa],
SSIM [wang2004ssim]) and likeness (identity consistency) (2) general image
quality (FID [heusel2017fid]) and (3) 3D reconstruction quality (depth, and
pose estimation). For the reconstruction tasks, we need to re-render our
outputs to the input views for the purpose of the evaluation using a camera
pose estimated using an off-the-shelf pose predictor [deng2019accurate].
However, we noticed that errors present in the estimated poses create a small
image misalignment between the ground truth and our feedforward results (as
opposed to inversion models which directly optimize for the given view),
making the raw pixel metrics like PSNR and SSIM unreliable. For this reason,
we mainly rely on the deep perceptual image metrics such as LPIPS and DISTS,
which judge that the given images are of the same perceptual quality for our
evaluation. Nonetheless, we report SSIM results in the main paper and include
PSNR results in the supplement along with an analysis of alignment issues. In
the end, our experiments qualitatively and quantitatively support that our
method achieves the state-of-the-art results on in-the-wild portraits as well
as multiview 3D scan datasets. For more results, please refer to the
supplement video.

#### Datasets.

Our method is evaluated on FFHQ [karras2019style], a representative dataset
for high-quality in-the-wild human portraits, H3DS [ramon2021h3d], which has
high resolution ground truth 3D scans and 360º images of 23 human heads with
associated camera calibrations, and AFHQv2 Cats [choi2020starganv2,
Karras2021], a collection of high-resolution in-the-wild portraits of cats.

###  5.1 Comparisons

#### Baselines.

We compare our methods against three state-of-the art methods for 3D aware-
image generation from a single image: ROME [Khakhulin2022ROME], HeadNeRF
[hong2021headnerf], and EG3D-PTI, which combines an unconditional EG3D
generator [eg3d2022] and Pivotal Tuning Inversion (PTI) [roich2021pivotal]. We
also compare with EG3D itself as an unconditional reference on FID. We
additionally provide extensive evaluations on our lightweight model (LT),
which is introduced in Sec. 4.

#### Qualitative results.

Fig. 1 shows our qualitative results on FFHQ and AFHQ. Fig. 5 and Fig. 6 show
selected examples from FFHQ, demonstrating high-quality novel views and 3D
geometry reconstructed by our method from a single portrait. Fig. 7 provides a
qualitative comparison against baselines. While HeadNeRF and ROME provide
adequate shapes and images, they need image segmentation as a preprocess, and
struggle with obtaining photorealistic results. Despite the 20 mins of fine
tuning, EG3D-PTI does not ensure the reconstruction looks photorealistic when
viewed from a non-input view (see Fig. 2). In contrast, our method
reconstructs the entire portrait with accurate photorealistic details. Fig. 8
provides comparisons to the ground truth validation view and 3D scan on H3DS.
The synthesized image and 3D geometry of ROME and HeadNeRF generally lack the
fidelity and reconstruct only a part of the head. EG3D-PTI occasionally
outputs a degenerate 3D shape due to the highly unconstrained nature of
single-view training of the NeRF representation (see Figs 1, 7 and 8). Our
geometry retains overall the 3D shape as well as person-specific facial
details. We also provide results on lifting 2D drawings and paintings into 3D
in Fig. 9. While our method is never trained with stylized images, it can
reasonably well handle those out-of-domain input images. Finally, we also show
the outputs of our method in comparison to baselines at varying pitch and yaw
in Figs 10 and 11, displaying the benefit of our method for photorealistic
facial frontalization of challenging images. In comparison to baselines, our
method’s geometry does not collapse for challenging yaws as EG3D-PTI, and
shows a significantly higher degree of photorealism than ROME and HeadNeRF.

Table 2: Quantitative evaluation using LPIPS, DISTS, SSIM, pose accuracy (Pose) and identity consistency (ID) on 500 FFHQ images. ${\dagger}$Evaluated only using the foreground on $256^{2}$ images. ${\ddagger}$Evaluated only using the face region. |  LPIPS $\\!\downarrow$ |  DISTS $\\!\downarrow$ |  SSIM $\\!\uparrow$ |  Pose $\\!\downarrow$ |  ID $\\!\uparrow$
---|---|---|---|---|---
HeadNeRF‡ | .2502 | .2427 | .7514 | .0644 | .2031
Ours‡ | .1240 | .0770 | .8246 | .0490 | .5481
ROME (256)† | .1158 | .1058 | .8257 | .0637 | .3231
Ours† | .0468 | .0407 | .8981 | .0486 | .5410
EG3D-PTI | .3236 | .1277 | .6722 | .0575 | .4650
Ours | .2692 | .0904 | .6598 | .0485 | .5426
Ours (LT) | .2750 | .1021 | .6655 |  .0448 | .5404

#### Quantitative evaluations.

Tab. 2 shows numerical comparisons of our method against baselines on 500
randomly selected images from FFHQ. We measure the 2D image reconstruction
quality in the input view using LPIPS, DISTS, and SSIM. We evaluate multiview
consistency using poses (Pose) estimated from synthesized images by an off-
the-shelf pose detector [deng2019accurate] following similar protocols as in
previous work [eg3d2022, shi2021lifting], and identity (ID) consistency by
computing the mean of MagFace [meng2021magface] (not used in our training)
cosine similarity scores between the input view and synthesized view from a
random camera pose. Since HeadNeRF and ROME only produce the face region and
the foreground respectively, we also provide the same metrics from our models
evaluated on the same masked region. Tab. 2 shows that our model significantly
outperforms the baselines on all the metrics except SSIM; our SSIM score is
only marginally lower than EG3D-PTI despite the aforementioned issue of the
image misalignment and the fact that EG3D-PTI directly optimizes the pixels
for the evaluation view. The geometry evaluation in Tab. 3 on H3DS in which we
compare the depths of the ground truth from the input view as predicted by
each model validates that our models produce more accurate 3D geometry.

Table 3: Scale- and translation-invariant depth evaluation using ground truth geometry from H3DS datasets. †Evaluated only using the face region. Depth | H.NeRF | ROME | EG3D-PTI | Ours | Ours (LT)
---|---|---|---|---|---
L1 $\\!\downarrow$ |  0.108† | 0.054 | 0.071 | 0.048 | 0.049
RMSE $\\!\downarrow$ |  0.147† | 0.084 | 0.101 | 0.074 | 0.075

###  5.2 Ablation study

We provide ablation studies comparing variants of our architecture and
different training strategies. All variants are evaluated after training with
3M images.

#### Inference time and number of parameters

We compare the performance of two variants of our model, which have the same
architecture but have different numbers of parameters and resolution of
intermediate feature maps: ”Ours” (87M params) and ”Ours (LT)” (63M params).
Tab. 1 provide runtime comparisons of the two. Tabs. 2 and 3 provide several
comparisons of the two on image reconstruction, the accuracy of 3D shapes and
identity consistencies. These extensive evaluations suggest that our
lightweight model retains very close performance to our full model despite
running significantly faster. Figs. 1 and 5 show qualitative samples from both
our full model and our lightweight model.

#### Effects of Transformers

Fig. 12 compares results obtained with or without the proposed Transformer
layers in the encoder. For this variant, we replaced the ViT module with CNN
with matching number of parameters. Tab. 4 provides numerical comparisons of
the two variants on image and 3D quality metrics. These quantitative and
qualitative comparisons show that the ViT layers are important for creating
more accurate 3D representations as well as achieving more accurate 2D image
reconstruction.

#### Effects of camera augmentation

Fig. 13 compares the models trained with or without the camera augmentation
for robustness to camera noise (also see the first row of Fig. 7 for the
results of the same subject without the camera noise). We fix the camera
calibration and apply image space rotation, translation, and zoom to the input
image, emulating the effect of inaccurate camera extrinsics and intrinsics.
Although our model does not rely on any camera information for
canonicalization, the result is not robust without the proposed camera
augmentation. EG3D-PTI assumes a fixed image alignment used to train the GAN
model and is very sensitive to small image misalignment in the input. Tab. 4
provides numerical comparisons of our model with and without the proposed
augmentation.

#### Effects of fine-tuned synthetic discriminator.

We provide an additional ablation on the discriminator loss ($L_{\text{adv}}$
in Eqn. 5), which fine-tunes the pre-trained EG3D discriminator with
EG3D-generated images. As seen in Tab. 5, removing this discriminator loss
results in a worse FID score. Moreover, as seen in Fig. 14, the renderings of
our proposed method are significantly sharper with the synthetic discriminator
tuning. Please see Sec. A1 and Tab. A2 for attempts to train the discriminator
with real images.

Fig. 12. Ablation study comparing our model with and without the proposed
Transformer modules. The model w/o Transformer replaces all Transformer Blocks
with resolution-preserving residual CNNs with similar parameters. Credit to
Kirill Chebotar.

Fig. 13. Camera augmentation ablation study. Note that this is the same image
as the first row of Fig. 7 except rotated and cropped non-centrally. Without
augmentation, our result exhibits artifacts when the input image has zoom or
camera roll. Similarly, EG3D-PTI is also sensitive to the image misalignment,
as the camera pose becomes noisy, while our method correctly canonicalizes the
face. Images are cropped and aligned for visual consistency. Credit to U.S.
Dept. of HUD.

Fig. 14. Comparison between our model and an ablated model trained without the
synthetic discriminator. Note the blurriness without the adversarial loss.
Credit to Mohd Fazlin Mohd Effendy Ooi.

Table 4: Ablation studies evaluating the proposed camera augmentation and the Transformer module. Without augmentation, the model acts as an autoencoder, mapping real images to arbitrary 3D representations that resemble the input (good ID score), but are not actually 3D (poor Pose score). Without a transformer, the encoder can canonicalize the inputs well (good Pose score), but cannot represent the details of the input (poor ID score). Our full method achieves both good Pose and ID scores with high reconstruction quality.  |  LPIPS $\\!\downarrow$ |  DISTS $\\!\downarrow$ |  Pose $\\!\downarrow$ |  ID $\\!\uparrow$ |  FID $\\!\downarrow$
---|---|---|---|---|---
No aug. | 0.3846 | .1286 | 0.1758 | 0.5359 | 3.42
No Transformer | 0.5419 | .1650 | 0.0426 | 0.1906 | 11.5
Ours | 0.2894 | .1053 | 0.0461 | 0.5230 | 4.45
Table 5: Comparison in FID between our model and an ablated model without the synthetic discriminator. FID $\\!\downarrow$ | FFHQ
---|---
w/o synthetic disc. | 7.71
Ours | 4.45

###  5.3 Application: real-time 3D telepresence

We apply our method for lifting a monocular RGB video input to 3D in real-
time, as would be needed for 3D telepresence. Our method processes the video
frame by frame. Despite being trained on individual frames of synthetic data
and processing the input video in a frame to frame fashion, our method can
provide reasonable temporal consistency. Please refer to the teaser Fig. 1
(bottom right) for the output from our lightweight model as well as video
examples from the supplement. Fig. 15 shows our system set up and running off
of a desktop with a single RTX 4090. Our method can lift a monocular RGB video
frame from a mobile phone to 3D in real-time.

Figure 15: Our system applied to create a 3D telepresence live from a
monocular RGB input. Please see the supplement video for the live
demonstration.

##  6 Discussion

#### Limitation.

When the input is a strong profile view (e.g., 60 degrees yaw angle), our
method may struggle with properly canonicalizing the input, as it is highly
out-of-distribution with respect to EG3D-generated images and FFHQ. Please see
Figs. 10 and 11 for various challenging levels of pitch and yaw for input
images. While our method can predict a canonicalized 3D representation without
requiring camera poses as input, the rendered image may be slightly misaligned
when compared to the input view (see Fig. A8 in the supplement for the
detailed analysis) possibly due to the combination of the imperfect
canonicalization and noisy camera poses from an off-the-shelf pose estimator.
Finally, although our method can provide reasonable temporal consistency when
applied to a video in a frame-by-frame fashion, temporal inconsistencies
remain as the canonicalizations change slightly per frame, and the predicted
camera poses are entirely independent.

#### Future work.

In the future, combining our method with camera pose optimization
[ko20233d3dganinversion] may lead to more accurate 3D reconstruction and
camera pose estimation. Additionally, jointly predicting the camera poses and
triplanes in an autoregressive or recurrent context
[srivastava2015unsupervised, kalchbrenner2017video, shi2015convolutional] may
result in more consistent frame-by-frame results. Next, it would be
interesting to incorporate real images in the training as our preliminary
attempts did not yield improvements. Finally, as our pipeline does not
necessarily assume any category-specific priors, we can view it as a general
method to distill the knowledge of a 3D GAN into a feedforward encoder. Thus,
extending 3D GANs to more general scenes [imagenet3d] may allow our pipeline
to create 3D representations of arbitrary scenes in the future. Specifically
extending our work to handle hands or the full body, is of interest for real-
time telepresence applications.

#### Conclusion.

We proposed a one-shot encoder-based framework to lift a single RGB image to
3D in real-time and demonstrated our method, trained entirely from synthetic
data, can handle challenging (even out-of-domain) real-world images. We
believe that this opens up possibilities for accessible 3D reconstructions of
real-world objects and interactive 3D visualization from a picture.

## Acknowledgements

We thank David Luebke, Jan Kautz, Peter Shirley, Alex Evans, Towaki Takikawa,
and Ekta Prashnani for feedback on drafts and early discussions. This work was
funded in part at UCSD by ONR grant N000142012529, an NSF graduate Fellowship,
a Jacobs Fellowship, and the Ronald L. Graham chair. Manmohan Chandraker
acknowledges support of NSF IIS 2110409. Koki Nagano and Eric Chan were
partially supported by DARPA’s Semantic Forensics (SemaFor) contract
(HR0011-20-3-0005). The views and conclusions contained in this document are
those of the authors and should not be interpreted as representing the
official policies, either expressed or implied, of the U.S. Government.
Distribution Statement “A” (Approved for Public Release, Distribution
Unlimited).