# Portrait Neural Radiance Fields from a Single Image

Chen Gao
Virginia Tech
Yichang Shih
Google
Wei-Sheng Lai
Google
Chia-Kai Liang
Google
Jia-Bin Huang
Virginia Tech

###### Abstract

We present a method for estimating Neural Radiance Fields (NeRF) from a single
headshot portrait. While NeRF has demonstrated high-quality view synthesis, it
requires multiple images of static scenes and thus impractical for casual
captures and moving subjects. In this work, we propose to pretrain the weights
of a multilayer perceptron (MLP), which implicitly models the volumetric
density and colors, with a meta-learning framework using a light stage
portrait dataset. To improve the generalization to unseen faces, we train the
MLP in the canonical coordinate space approximated by 3D face morphable
models. We quantitatively evaluate the method using controlled captures and
demonstrate the generalization to real portrait images, showing favorable
results against state-of-the-arts.

https://portrait-nerf.github.io

$\underbracket{\hskip 84.55907pt}_{\begin{subarray}{c}\vspace{-2.0mm}\\\
\hbox{\pagecolor{white}(a) Input}\end{subarray}}$ $\underbracket{\hskip
169.11153pt}_{\begin{subarray}{c}\vspace{-2.0mm}\\\ \hbox{\pagecolor{white}(b)
Novel view synthesis}\end{subarray}}$ $\underbracket{\hskip
169.11153pt}_{\begin{subarray}{c}\vspace{-2.0mm}\\\ \hbox{\pagecolor{white}(c)
FOV manipulation}\end{subarray}}$

Figure 1:  Applications of the proposed method. Given a single portrait image
(a) as input, our method produces a _portrait neural radiance field_ that
facilitates photorealistic face editing tasks such as (b) novel view synthesis
and (c) field-of-view (FOV) manipulation, where the left and right are
rendered with wider and narrower camera FOVs, respectively.

##  1 Introduction

Portrait view synthesis enables various post-capture edits and computer vision
applications, such as pose manipulation [10], selfie perspective distortion
(foreshortening) correction [82, 15, 42], improving face recognition accuracy
by view normalization [83], and greatly enhancing the 3D viewing experiences.
Compared to 3D reconstruction and view synthesis for generic scenes, portrait
view synthesis holds a higher quality standard to avoid the uncanny valley.

To achieve high-quality view synthesis, the filmmaking production industry
densely samples lighting conditions and camera poses using a light stage [11],
which requires expensive hardware setups unsuitable for casual users.
Reconstructing the facial geometry from a _single capture_ requires face mesh
templates [6] or a 3D morphable model [4, 7, 5, 31]. While the quality of
these 3D model-based methods has been improved dramatically via deep networks
[18, 74], a common limitation is that the model only covers the center of the
face and excludes the upper head, hairs, and torso. We emphasize that the
excluded regions are critical for natural portrait view synthesis.

Recently, neural implicit representations emerge as a promising way to model
the appearance and geometry of 3D scenes and objects [58, 41, 35]. For
example, Neural Radiance Fields (NeRF) demonstrates high-quality view
synthesis by implicitly modeling the volumetric density and color using the
weights of a multilayer perceptron (MLP). However, training the MLP requires
capturing images of _static subjects_ from _multiple viewpoints_ (in the order
of 10-100 images) [41, 38]. It is thus impractical for portrait view synthesis
because a slight subject movement or inaccurate camera pose estimation
degrades the reconstruction quality.

In this paper, we propose to train an MLP for modeling the radiance field
using a _single headshot portrait_ illustrated in Figure 1. Unlike NeRF [41],
training the MLP with a single image from scratch is fundamentally ill-posed,
because there are infinite solutions where the renderings match the input
image. Our key idea is to pretrain the MLP and finetune it using the available
input image, and adapt the model to an unseen subject’s appearance and shape.
To pretrain the MLP, we use densely sampled portrait images in a light stage
capture. However, using a naïve pretraining process that optimizes the
reconstruction error between the synthesized views (using the MLP) and the
rendering (using the light stage data) over the subjects in the dataset
performs poorly for _unseen_ subjects due to the diverse appearance and shape
variations among humans. We address the challenges in two novel ways. First,
we leverage gradient-based meta-learning techniques [13] to train the MLP that
quickly adapts to an unseen subject. Second, we propose to train the MLP in a
canonical coordinate by exploiting domain-specific knowledge about the face
shape. Our experiments show favorable quantitative results against the state-
of-the-art 3D face reconstruction and synthesis as well as 2D GAN-based
algorithms on the dataset of controlled captures. We show $+5$dB PSNR
improvement against the second best approach in ground truth validation. We
further show that our method performs well for real input images captured in
the wild and demonstrate foreshortening distortion correction as an
application.

In this work, we make the following contributions:

  * •

We present a single-image view synthesis algorithm for portrait photos by
leveraging meta-learning. Our method produces a _full reconstruction_ ,
covering not only the facial area but also the upper head, hairs, torso, and
accessories such as eyeglasses.

  * •

We propose an algorithm to pretrain NeRF in a canonical face space using a
rigid transform from the world coordinate. We show that compensating the shape
variations among the training data substantially improves the model
generalization to unseen subjects.

  * •

We provide a multi-view portrait dataset consisting of controlled captures in
a light stage. Our data provide a way of quantitatively evaluating portrait
view synthesis algorithms.

##  2 Related Work

\begin{overpic}[width=433.62pt]{fig/method/overview_v4.pdf}
\put(30.0,3.0){${\scriptstyle(\vec{x},\vec{d})}$}
\put(44.0,3.0){${\scriptstyle(\mathit{s}\mathbf{R}\vec{x}+\mathbf{t},\vec{d})}$}
\put(62.0,13.7){${\scriptstyle(\mathit{s}\mathbf{R}\vec{x}+\mathbf{t},\vec{d})}$}
\put(25.5,7.0){${\scriptstyle\theta}$} \put(54.5,7.0){${\scriptstyle
f_{\theta}}$}
\put(25.0,14.0){\rotatebox[origin={c}]{90.0}{${\scriptstyle(\vec{x},\vec{d})}$}}
\put(0.5,7.7){${\scriptstyle\theta_{p,m}}$}
\par\par\put(77.0,7.7){${\scriptstyle\theta_{s}}$} \end{overpic}

(a) Pretrain NeRF (b) Warp to canonical coordinate (c) Finetune

Figure 2: Overview. Our method builds on the MLP network $f_{\theta}$ in NeRF
[41], and requires only a single image as input at the test time. To learn the
geometry and shape priors for single-image view synthesis, we (a) pretrain the
model parameter $\theta_{p}$ using the light stage dataset (Section 3.1)
consisting of multiple training views and subjects indexed by $m$ (Section
3.2). To improve the generalization, we (b) warp the world coordinate
$\vec{x}$ to the canonical coordinate derived from the 3D model through a
rigid transform $(s,\mathbf{R},\mathbf{t})$, and feed forward the warped
coordinate and viewing direction $\vec{d}$ to $f_{\theta}$ (Section 3.3)
during the view synthesis process. In test time, (c) we finetune the model
parameter $\theta_{s}$ against the input view, and use $f_{\theta_{s}}$ and
volume rendering from color and occupancy ($RGB\sigma$) for view synthesis
(Section 3.4).

View synthesis with neural implicit representations. Our method builds on
recent work of neural implicit representations [58, 41, 36, 79, 3, 38, 73] for
view synthesis. NeRF [41] represents the scene as a mapping $\mathcal{F}$ from
the world coordinate and viewing direction to the color and occupancy using a
compact MLP. Given a camera pose, one can synthesize the corresponding view by
aggregating the radiance over the light ray cast from the camera pose using
standard volume rendering. The MLP is trained by minimizing the reconstruction
loss between synthesized views and the corresponding ground truth input
images. Existing methods require tens to hundreds of photos to train a scene-
specific NeRF network. In contrast, our method requires only _one single
image_ as input. We finetune the pretrained weights learned from light stage
training data [11, 39] for unseen inputs. Compared to the unstructured light
field [40, 14, 52, 48], volumetric rendering [37, 45], and image-based
rendering [22, 21], our single-image method does not require estimating camera
pose [53], which is challenging at the complex structures and view-dependent
properties, like hairs and subtle movement of the subjects between captures.
Similar to the neural volume method [37], our method improves the rendering
quality by sampling the warped coordinate from the world coordinates.

Existing single-image view synthesis methods model the scene with point cloud
[46, 70], multi-plane image [66, 24], or layered depth image [55, 29]. Our
method focuses on headshot portraits and uses an implicit function as the
neural representation. We demonstrate foreshortening correction as
applications [82, 15, 42]. Instead of training the warping effect between a
set of _pre-defined_ focal lengths [82, 42], our method achieves the
perspective effect at _arbitrary_ camera distances and focal lengths.

Face pose manipulation. Conditioned on the input portrait, generative methods
learn a face-specific Generative Adversarial Network (GAN) [20, 26, 27] to
synthesize the target face pose driven by exemplar images [72, 49, 47, 64, 28,
78, 68], rig-like control over face attributes via face model [62, 17, 19, 30,
54, 77], or learned latent code [12, 1]. While these methods yield impressive
results and the outputs are photorealistic, these approaches have common
artifacts that the generated images often exhibit inconsistent facial
features, identity, hairs, and geometries across the results and the input
image. Our method faithfully preserves the texture and geometry information of
the subject across camera poses by using the 3D neural representation
invariant to camera poses [63, 44] and taking advantage of pose-supervised
training [75].

3D face modeling. Reconstructing face geometry and texture enables view
synthesis using graphics rendering pipelines. Existing single-view methods use
the symmetric cues [71], morphable model [4, 7, 5, 31, 8], mesh template
deformation [6], regression with deep networks [25], and videos [59]. However,
these _model-based_ methods only reconstruct the regions where the model is
defined, and do not handle hairs and torsos, or require a separate hair
modeling post-processing [74, 23, 32]. Our method combines the benefits from
both face-specific modeling and view synthesis on generic scenes. We leverage
the face domain-specific knowledge by training on a portrait dataset and
propose the canonical face coordinates using the 3D face geometric proxy
derived by a morphable model. To model the portrait subject, instead of using
facial landmark mesh, we use the finetuned NeRF at the test time to include
hairs and torsos.

Meta-learning. Our work is closely related to meta-learning and few-shot
learning [51, 2, 13, 9, 60, 65]. To explain the analogy, we consider view
synthesis from a camera pose as a query, captures associated with the known
camera poses from the light stage dataset as labels, and training a subject-
specific NeRF as a task. Training NeRFs for different subjects is analogous to
training classifiers for various tasks. At the test time, given a single label
from the frontal capture, our goal is to optimize the testing task, which
learns the NeRF to answer the queries of camera poses. We leverage gradient-
based meta-learning algorithms [13, 57] to learn the weight initialization for
the MLP in NeRF from the meta-training tasks, _i.e_., learning a single NeRF
for different subjects in the light stage dataset.

Concurrent work on neural volumetric rendering. Several concurrent efforts
have been proposed to use neural volumetric rendering for modeling photo-
realistic human heads [50, 16, 69]. These methods support synthesizing human
heads from a wider range of viewpoints but still require multiple input views.
Recent work also extends NeRF for handling single image input using meta-
learning [61] or pixel-aligned representation [76]. Our work focuses on head
portrait shots and shows that our use of canonical face coordinates (guided by
the fixed 3D morphable model) greatly improves the model generalization to
unseen faces.

##  3 Algorithm

Figure 2 provides an overview of our method, consisting of pretraining and
testing stages. In the pretraining stage, we train a coordinate-based MLP
(same in NeRF) $f_{\theta}$ on diverse subjects captured from the light stage
and obtain the pretrained model parameter optimized for generalization to
unseen subjects, denoted as $\theta_{p}^{*}$ (Section 3.2). The high
diversities among the real-world subjects in identities, facial expressions,
and face geometries are challenging for training. We address the face shape
variation by deforming the world coordinate to the canonical face coordinate
using a rigid transform, and train a shared representation across the subjects
on the normalized coordinate (Section 3.3). At the test time, we initialize
the NeRF with our pretrained model parameter $\theta_{p}^{*}$ and then
finetune it on the frontal view for the input subject $s$. We use the
finetuned model parameter (denoted by $\theta_{s}^{*}$) for view synthesis
(Section 3.4).

###  3.1 Training data

Our training data consists of light stage captures over multiple subjects.
Each subject is lit uniformly under controlled lighting conditions. We process
the raw data to reconstruct the depth, 3D mesh, UV texture map, photometric
normals, UV glossy map, and visibility map for the subject [81, 39]. We render
a sequence of 5-by-5 training views for each subject by uniformly sampling the
camera locations over a solid angle centered at the subject’s face at a fixed
distance between the camera and subject. We span the solid angle by 25° field-
of-view vertically and 15° horizontally. We set the camera viewing directions
to look straight to the subject. Figure 3 and supplemental materials show
examples of 3-by-3 training views. The center view corresponds to the front
view expected at the test time, referred to as the support set
$\mathcal{D}_{s}$, and the remaining views are the target for view synthesis,
referred to as the query set $\mathcal{D}_{q}$. We refer to the process
training a NeRF model parameter for subject $m$ from the support set as a
task, denoted by $\mathcal{T}_{m}$.

\begin{overpic}[width=433.62pt]{fig/method/pretrain_v5.pdf}
\put(19.0,9.0){$\theta_{p,m}$} \put(39.0,12.0){Updates by ~{}\eqref{eq:inner}}
\put(58.0,8.0){$\theta_{m}^{*}$} \put(75.0,12.0){Updates by
~{}\eqref{eq:update-init-inner-1}} \put(63.0,2.0){Updates by
~{}\eqref{eq:update-init-inner}} \put(88.0,6.0){$\theta_{p,m+1}$}
\par\end{overpic} Figure 3:  Pretraining with meta-learning framework. The
training views of each subject $m$ consists of the frontal view, called
support set $D_{s}$, and the rest views, called query set $D_{q}$. In each
iteration, we update the pretrained parameter from $\theta_{p,m}$ to
$\theta_{p,m+1}$ using two steps. We first update the parameter to
$\theta_{m}^{*}$ by finetuning on the $D_{s}$ using (1). Then we continue to
update $\theta_{m}^{*}$ using (2) on $D_{q}$, and feedback the update to the
pretrained parameter $\theta_{p,m+1}$ using (3).

###  3.2 Pretraing NeRF

Our goal is to pretrain a NeRF model parameter $\theta_{p}^{*}$ that can
easily adapt to capturing an _unseen_ subject’s appearance and geometry. We
loop through $K$ subjects in the dataset, indexed by $m=\\{0,...,K-1\\}$, and
denote the model parameter pretrained on the subject $m$ as $\theta_{p,m}$. We
sequentially train on subjects in the dataset and update the pretrained model
as $\\{\theta_{p,0},\theta_{p,1}...,\theta_{p,K-1}\\}$, where the last
parameter is outputted as the final pretrained model,  _i.e_.,
$\theta_{p}^{*}=\theta_{p,K-1}$. For each task $\mathcal{T}_{m}$, we train the
model on $\mathcal{D}_{s}$ and $\mathcal{D}_{q}$ alternatively in an inner
loop, as illustrated in Figure 3. Since $\mathcal{D}_{s}$ is available at the
test time, we only need to propagate the gradients learned from
$\mathcal{D}_{q}$ to the pretrained model $\theta_{p}^{*}$, which transfers
the common representations unseen from the front view $\mathcal{D}_{s}$ alone,
such as the priors on head geometry and occlusion.

Pretraining on $\mathcal{D}_{s}$. For the subject $m$ in the training data, we
initialize the model parameter from the pretrained parameter learned in the
previous subject $\theta_{p,m-1}$, and set $\theta_{p,-1}$ to random weights
for the first subject in the training loop. We train a model $\theta_{m}^{*}$
optimized for the front view of subject $m$ using the $L_{2}$ loss between the
front view predicted by $f_{\theta_{m}}$ and $\mathcal{D}_{s}$ , denoted as
$\mathcal{L}_{\mathcal{D}_{s}}(f_{\theta_{m}})$. The optimization iteratively
updates the $\theta_{m}^{t}$ for $N_{s}$ iterations as the following:

$\displaystyle\theta_{m}^{t+1}=\theta_{m}^{t}-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{D}_{s}}(f_{\theta_{m}^{t}}),$
(1)

where $\theta_{m}^{0}=\theta_{p,m-1}$, $\theta_{m}^{*}=\theta_{m}^{N_{s}-1}$,
and $\alpha$ is the learning rate.

Pretraining on $\mathcal{D}_{q}$. We proceed to update the pretrained
parameter $\theta_{p,m}$ using the loss between the prediction from the known
camera pose and the query dataset unseen in test time, as described in the
following:

$\displaystyle\theta_{m}^{t+1}=\theta_{m}^{t}-\beta\nabla_{\theta}\mathcal{L}_{\mathcal{D}_{q}}(f_{\theta_{m}^{t}}),$
(2)
$\displaystyle\theta_{p,m}^{t+1}=\theta_{p,m}^{t}-\beta\nabla_{\theta}\mathcal{L}_{\mathcal{D}_{q}}(f_{\theta_{m}^{t}}),$
(3)

where $\theta_{m}^{0}=\theta_{m}^{*}$ learned from $D_{s}$ in (1),
$\theta_{p,m}^{0}=\theta_{p,m-1}$ from the pretrained model on the previous
subject, and $\beta$ is the learning rate. After $N_{q}$ iterations, we update
the pretrained parameter by the following:

$\displaystyle\theta_{p,m}=\theta_{p,m}^{N_{q}-1}$ (4)

Note that (3) does not affect the update of the current subject $m$, _i.e_.,
(2), but the gradients are propagated to the next training subject in the
subsequent iterations through the pretrained model parameter update in (4).
The training is terminated after visiting the entire dataset over $K$
subjects. The pseudo-code of the algorithm is described in the supplemental
material.

Discussion. We assume that the order of applying the gradients learned from
$\mathcal{D}_{q}$ and $\mathcal{D}_{s}$ are interchangeable, similarly to the
first-order approximation in MAML algorithm [13]. We transfer the gradients
from $\mathcal{D}_{q}$ independently of $\mathcal{D}_{s}$. For better
generalization to unseen subjects, the gradients of $\mathcal{D}_{s}$ will be
adapted from the input subject at the test time by finetuning, instead of
transferred from the training data. In our experiments, applying the meta-
learning algorithm designed for image classification [65] performs poorly for
view synthesis. This is because each update in view synthesis requires
gradients gathered from _millions of samples_ across the scene coordinates and
viewing directions, which do not fit into a single batch in modern GPU. Our
method takes a lot more steps in a single meta-training task for better
convergence.

###  3.3 Canonical face space

(a) Subject

(b) World coordinate

(c) Canonical face coordinate

(d) Mean face

Figure 4:  Rigid transform between the world and canonical face coordinate.
For every subject (a), we detect the face mesh (b), and fit a rigid transform
(c) using the vertices correspondence between (b) and the mean face of the
dataset (d). We warp from the world (b) to the canonical (c) coordinate, and
feed the warped coordinate to NeRF to predict color and occupancy. The meshes
(b-d) are for visualization and not used in view synthesis. Only the rigid
transform is used in our work.

(a) Input (front view)

(b) Top-left

(c) Bottom-left

(d) Bottom

(e) Top-Right

(f) Right

(g) Disparity on (a)

Figure 5: View synthesis from a single front view. Our method finetunes the
pretrained model on (a), and synthesizes the new views using the controlled
camera poses (c-g) relative to (a). The results in (c-g) look realistic and
natural. To validate the face geometry learned in the finetuned model, we
render the (g) disparity map for the front view (a). The videos are
accompanied in the supplementary materials.

To address the face shape variations between the training dataset and real-
world inputs, we normalize the world coordinate to the canonical space using a
rigid transform and apply $f_{\theta}$ on the warped coordinate. Specifically,
for each subject $m$ in the training data, we compute an approximate facial
geometry $F_{m}$ from the frontal image using a 3D morphable model and image-
based landmark fitting [7]. We average all the facial geometries in the
dataset to obtain the mean geometry $\overline{F}$. During the training, we
use the vertex correspondences between $F_{m}$ and $\overline{F}$ to optimize
a rigid transform by the SVD decomposition (details in the supplemental
documents). The transform is used to map a point $\mathbf{x}$ in the subject’s
world coordinate to $\mathbf{x^{\prime}}$ in the face canonical space:
$\mathbf{x^{\prime}}=s_{m}\mathbf{R}_{m}\mathbf{x}+\mathbf{t}_{m}$, where
$s_{m},\mathbf{R}_{m}$ and $\mathbf{t}_{m}$ are the optimized scale, rotation,
and translation minimizing the $L_{2}$ loss between vertices of $F_{m}$ and
$\overline{F}$.

During the prediction, we first warp the input coordinate from the world
coordinate to the face canonical space through
$(s_{m},\mathbf{R}_{m},\mathbf{t}_{m})$. We then feed the warped coordinate to
the MLP network $f_{\theta}$ to retrieve color and occlusion (Figure 4). We
keep the viewing direction $\vec{d}$ in the world coordinate. The warp makes
our method robust to the variation in face geometry and poses in the training
and testing inputs, as shown in Table 3 and Figure 10. In our method, the 3D
model is used to obtain the rigid transform
$(s_{m},\mathbf{R}_{m},\mathbf{t}_{m})$. We do _not_ require the mesh details
and priors as in other model-based face view synthesis [74, 7].

###  3.4 Finetuning and rendering

At the test time, only a single frontal view of the subject $s$ is available.
We first compute the rigid transform described in Section 3.3 to map between
the world and canonical coordinate. Then, we finetune the pretrained model
parameter $\theta_{p}^{*}$ by repeating the iteration in (1) for the input
subject, and output the optimized model parameter $\theta_{s}^{*}$. To render
novel views, we sample the camera ray in the 3D space, warp to the canonical
space, and feed to $f_{\theta_{s}^{*}}$ to retrieve the radiance and occlusion
for volume rendering.

##  4 Experimental Results

$\underbracket{\hskip 84.55907pt}_{\begin{subarray}{c}\vspace{-2.0mm}\\\
\hbox{\pagecolor{white}Input}\end{subarray}}$ $\underbracket{\hskip
169.11153pt}_{\begin{subarray}{c}\vspace{-2.0mm}\\\ \hbox{\pagecolor{white}Our
method}\end{subarray}}$ $\underbracket{\hskip
169.11153pt}_{\begin{subarray}{c}\vspace{-2.0mm}\\\
\hbox{\pagecolor{white}Ground truth}\end{subarray}}$

Figure 6: Comparisons to the ground truth. Our results faithfully preserve the
details like skin textures, personal identity, and facial expressions from the
input.

Input

Jackson  _et al_. [25]

Siarohin  _et al_. [56]

Wang  _et al_. [68]

Zakharov  _et al_. [77]

Xu  _et al_. [74]

Our method

Ground truth

Figure 7: Comparison to the state-of-the-art portrait view synthesis on the
light stage dataset. In each row, we show the input frontal view and two
synthesized views using [25, 74] and our method. The method by Jackson  _et
al_. [25] recovers the geometry only in the center of the face. The deep 3D
portrait [74] is capable of reconstructing the full head (including hair and
ear), but fails to preserve eye gaze, expression, face shape, identity, and
hairstyle when comparing to the ground truth. The face reenactment methods
[56, 68, 77] do not take the camera poses as input, thus a different person
with the same ground truth pose to drive the synthesis. The results either
lose identity or generate incorrect gaze direction. Our method produces
photorealistic results closest to the ground truth, as shown in the numerical
evaluations in Table 1.

Our dataset consists of 70 different individuals with diverse gender, races,
ages, skin colors, hairstyles, accessories, and costumes. For each subject, we
capture 2-10 different expressions, poses, and accessories on a light stage
under fixed lighting conditions. We include challenging cases where subjects
wear glasses. In total, our dataset consists of 230 captures. We hold out six
captures for testing. We render the support $\mathcal{D}_{s}$ and query
$\mathcal{D}_{q}$ by setting the camera field-of-view to 84°, a popular
setting on commercial phone cameras, and sets the distance to $30$cm to mimic
selfies and headshot portraits taken on phone cameras. Details about choosing
the parameters can be found in the supplemental materials.

Figure 5 shows our results on the diverse subjects taken in the wild. The
subjects cover different genders, skin colors, races, hairstyles, and
accessories. We stress-test the challenging cases like the glasses (the top
two rows) and curly hairs (the third row). Our results generally look
realistic, preserve the facial expressions, geometry, identity from the input,
handle the occluded area well, and successfully synthesize the clothes and
hairs for the subject. Our method generalizes well thanks to the finetuning
and canonical face coordinate, which closes the gap between the unseen
subjects and the pretrained model weights learned from the light stage
dataset. Figure 5 also shows that our method is robust when the input face
poses are slightly rotated away from the frontal view. In the supplemental
video, we hover the camera in the spiral path to demonstrate the 3D effect.
Our method preserves temporal coherence in challenging areas like hairs and
occlusion, such as the nose and ears.

Figure 6 compares our results to the ground truth using the subject in the
test hold-out set. Our method allows precise control of the camera pose and
faithfully reconstructs the details from the subject, as shown in the insets.

Table 1:  View synthesis metrics. We report the average PSNR, SSIM and LPIPS metrics against the existing methods over the testing set from our light stage dataset.  |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Jackson  _et al_. [25] | 10.23 | 0.4356 | 0.485
Wang  _et al_. [68] | 14.70 | 0.4578 | 0.380
Zakharov  _et al_. [77] | 15.25 | 0.4746 | 0.403
Siarohin  _et al_. [56] | 15.90 | 0.5149 | 0.437
Xu  _et al_. [74] | 18.91 | 0.5609 | 0.276
Our method | 23.92 | 0.7688 | 0.161

Comparisons. Figure 7 compares our method to the state-of-the-art face pose
manipulation methods [74, 25] and three face reenactment methods [56, 68, 77]
on six testing subjects held out from the training. The subjects cover various
ages, gender, races, and skin colors. The results from [74] were kindly
provided by the authors. We obtain the results of the remaining methods using
the official implementation with default parameters. The work by Jackson  _et
al_. [25] only covers the face area. The learning-based head reconstruction
method from Xu  _et al_. [74] generates plausible results but fails to
preserve the gaze direction, facial expressions, face shape, and hairstyles
(the bottom row) when comparing to the ground truth. Since the face
reenactment methods do not take the camera poses as input, we use a driving
face from a different person with the same ground truth pose for results.
While these face reenactment methods yield impressive results, they often lose
identity or generate incorrect gaze direction. Our results are objectively
closer to the ground truth based on PSNR, and covers the entire subject,
including hairs and body. We report the quantitative evaluation using PSNR,
SSIM, and LPIPS [80] against the ground truth in Table 1.

(a) Input

(b) Closer to subject

(c) Further

Figure 8: Perspective effect manipulation. Given an input (a), we virtually move the camera closer (b) and further (c) to the subject, while adjusting the focal length to match the face size. We manipulate the perspective effects such as dolly zoom in the supplementary materials.  Table 2:  Ablation study on initialization methods. Initialization |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Random | 14.99 | 0.5763 | 0.491
Pretrain by (5) | 22.87 | 0.7824 | 0.215
Our method | 23.70 | 0.8051 | 0.178
Table 3:  Ablation study on canonical face coordinate. Coordinate |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
World | 24.80 | 0.8167 | 0.172
Canonical (our method) | 24.98 | 0.8178 | 0.156

Perspective manipulation. Portraits taken by wide-angle cameras exhibit
undesired foreshortening distortion due to the perspective projection [15,
82]. As our representation is 3D in nature, we can render a new image by
virtually moving the camera closer (or further) from the subject and adjusting
the focal length correspondingly to preserve the face area. We demonstrate
perspective effect manipulation using portrait NeRF in Figure 8 and the
supplemental video. When the camera sets a longer focal length, the nose looks
smaller, and the portrait looks more natural. Since our training views are
taken from a single camera distance, the vanilla NeRF rendering [41] requires
inference on the world coordinates outside the training coordinates and leads
to the artifacts when the camera is too far or too close, as shown in the
supplemental materials. We address the artifacts by re-parameterizing the NeRF
coordinates to infer on the training coordinates.

(a) Random

(b) Pretrain by (5)

(c) Our method

(d) GT

Figure 9:  Ablation study on different weight initialization. (a) Training the
MLP from scratch using random weight initialization fails to converge and
leads to poor results. (b) Pretraining using (5) suffers from blurry rendering
and artifacts. (c) Our method produces the best quality compared to the ground
truth (d).

(a) Input

(b) World coordinate

(c) Our method

Figure 10: Ablation study on face canonical coordinates. Our method using (c)
canonical face coordinate shows better quality than using (b) world coordinate
on chin and eyes.

(a) 1 view

PSNR = 24.98

(b) 2 views

PSNR = 27.70

(c) 5 views

PSNR = 32.45

(d) GT

Figure 11: Ablation study on the number of input views during testing. Our results improve when more views are available.  Table 4:  Ablation study on training sizes. Training size |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
18 | 24.61 | 0.8110 | 0.163
27 | 24.98 | 0.8178 | 0.156
59 | 25.04 | 0.8129 | 0.164
100 | 24.65 | 0.8120 | 0.165
Table 5:  Ablation study on number of input views. | Random initialization |  |  Pretrain with (5) |  | Our method
---|---|---|---|---|---
Number of input views |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
1 | 17.34 | 0.6988 | 0.347 |  | 24.47 | 0.8109 | 0.175 |  | 24.98 | 0.8178 | 0.156
2 | 23.21 | 0.7936 | 0.214 |  | 27.37 | 0.8445 | 0.130 |  | 27.70 | 0.8647 | 0.115
5 | 31.12 | 0.8958 | 0.094 |  | 31.57 | 0.8826 | 0.107 |  | 32.45 | 0.9045 | 0.090

###  4.1 Ablation study

Initialization. Figure 9 compares the results finetuned from different
initialization methods. Without any pretrained prior, the random
initialization [41] in Figure 9(a) fails to learn the geometry from a single
image and leads to poor view synthesis quality. Next, we pretrain the model
parameter by minimizing the $L_{2}$ loss between the prediction and the
training views across all the subjects in the dataset as the following:

$\displaystyle\theta^{*}=\operatorname*{\arg\\!\min}_{\theta}\sum_{m}\mathcal{L}_{\mathcal{D}_{s}}(f_{\theta})+\mathcal{L}_{\mathcal{D}_{q}}(f_{\theta}),$
(5)

where $m$ indexes the subject in the dataset. Figure 9(b) shows that such a
pretraining approach can also learn geometry prior from the dataset but shows
artifacts in view synthesis. The synthesized face looks blurry and misses
facial details. Our pretraining in Figure 9(c) outputs the best results
against the ground truth. The quantitative evaluations are shown in Table 2.

Canonical face coordinate. Figure 10 and Table 3 compare the view synthesis
using the face canonical coordinate (Section 3.3) to the world coordinate.
Without warping to the canonical face coordinate, the results using the world
coordinate in Figure 10(b) show artifacts on the eyes and chins. Our method
outputs a more natural look on face in Figure 10(c), and performs better on
quality metrics against ground truth across the testing subjects, as shown in
Table 3.

Training task size. Our method does not require a large number of training
tasks consisting of many subjects. In Table 4, we show that the validation
performance saturates after visiting 59 training tasks. To balance the
training size and visual quality, we use 27 subjects for the results shown in
this paper. Compared to the majority of deep learning face synthesis works,
_e.g_., [74], which require _thousands of individuals_ as the training data,
the capability to generalize portrait view synthesis from a smaller subject
pool makes our method more practical to comply with the privacy requirement on
personally identifiable information.

Input views in test time. Our method can incorporate multi-view inputs
associated with known camera poses to improve the view synthesis quality. At
the finetuning stage, we compute the reconstruction loss between each input
view and the corresponding prediction. We show the evaluations on different
number of input views against the ground truth in Figure 11 and comparisons to
different initialization in Table 5. Compared to the vanilla NeRF using random
initialization [41], our pretraining method is highly beneficial when very few
(1 or 2) inputs are available. The margin decreases when the number of input
views increases and is less significant when 5+ input views are available.

###  4.2 Limitations

While our work enables training NeRF from a single portrait image, there are
several limitations. First, our method cannot handle the background, which is
diverse and difficult to collect on the light stage (Figure 12(a)). Users can
use off-the-shelf person segmentation [67] to separate the foreground, inpaint
the background [34], and composite the synthesized views to address the
limitation. Second, our method requires the input subject to be roughly in
frontal view and does not work well with the profile view (Figure 12(b)).
Extrapolating the camera pose to the unseen poses from the training data is
challenging and could lead to artifacts. Third, as we build on the volume
rendering in NeRF, our method is similarly slow at the runtime. We believe
that the slow rendering speed can be alleviated by integrating with recent
advances on speeding up volumeric rendering [33, 35, 43]. Fourth, our work
focuses only on a static image and leaves the extension to dynamic portrait
video to future work.

##  5 Conclusions

(a) Background

(b) Non-frontal view

Figure 12: Limitations. Left and right in (a) and (b): input and output of our
method. (a) When the background is not removed, our method cannot distinguish
the background from the foreground and leads to severe artifacts. (b) When the
input is not a frontal view, the result shows artifacts on the hairs.

We presented a method for portrait view synthesis using a single headshot
photo. Our method builds upon the recent advances of neural implicit
representation and addresses the limitation of generalizing to an unseen
subject when only one single image is available. Specifically, we leverage
gradient-based meta-learning for pretraining a NeRF model so that it can
quickly adapt using light stage captures as our meta-training dataset. We also
address the shape variations among subjects by learning the NeRF model in
canonical face space. We validate the design choices via ablation study and
show that our method enables natural portrait view synthesis compared with
state of the arts.

## References

  * [1] Yazeed Alharbi and Peter Wonka.  Disentangled image generation through structured noise injection.  In CVPR, 2020.
  * [2] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  In NeurIPS, 2016.
  * [3] Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel.  X-fields: Implicit neural view-, light-and time-image interpolation.  ACM TOG, 2020.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In CGIT, 1999.
  * [5] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway.  A 3d morphable model learnt from 10,000 faces.  In CVPR, 2016.
  * [6] Sofien Bouaziz, Yangang Wang, and Mark Pauly.  Online modeling for realtime facial animation.  ACM TOG, 2013.
  * [7] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  TVCG, 2013.
  * [8] Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou.  High-quality hair modeling from a single portrait photo.  ACM TOG, 2015.
  * [9] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  In ICLR, 2019.
  * [10] Antonio Criminisi, Jamie Shotton, Andrew Blake, and Philip HS Torr.  Gaze manipulation for one-to-one teleconferencing.  In ICCV, 2003.
  * [11] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar.  Acquiring the reflectance field of a human face.  In CGIT, 2000.
  * [12] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong.  Disentangled and controllable face image generation via 3d imitative-contrastive learning.  In CVPR, 2020.
  * [13] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  ICML, 2017.
  * [14] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In CVPR, 2019.
  * [15] Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein.  Perspective-aware manipulation of portrait photos.  ACM TOG, 2016.
  * [16] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  arXiv preprint arXiv:2012.03065, 2020.
  * [17] Baris Gecer, Binod Bhattarai, Josef Kittler, and Tae-Kyun Kim.  Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3d morphable model.  In ECCV, 2018.
  * [18] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T Freeman.  Unsupervised training for 3d morphable model regression.  In CVPR, 2018.
  * [19] Partha Ghosh, Pravir Singh Gupta, Roy Uziel, Anurag Ranjan, Michael Black, and Timo Bolkart.  Gif: Generative interpretable faces.  arXiv:2009.00149, 2020.
  * [20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [21] Peter Hedman and Johannes Kopf.  Instant 3d photography.  ACM TOG, 2018.
  * [22] Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow.  Deep blending for free-viewpoint image-based rendering.  ACM TOG, 2018.
  * [23] Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li.  Single-view hair modeling using a hairstyle database.  ACM TOG, 2015.
  * [24] Hsin-Ping Huang, Hung-Yu Tseng, Hsin-Ying Lee, and Jia-Bin Huang.  Semantic view synthesis.  In ECCV, 2020.
  * [25] Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos.  Large pose 3d face reconstruction from a single image via direct volumetric cnn regression.  In ICCV, 2017.
  * [26] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [27] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, 2020.
  * [28] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt.  Deep video portraits.  ACM TOG, 2018.
  * [29] Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, et al.  One shot 3d photography.  ACM TOG, 2020.
  * [30] Marek Kowalski, Stephan J Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, and Jamie Shotton.  Config: Controllable neural face image generation.  ECCV, 2020.
  * [31] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM TOG, 2017.
  * [32] Shu Liang, Xiufeng Huang, Xianyu Meng, Kunyao Chen, Linda G Shapiro, and Ira Kemelmacher-Shlizerman.  Video to fully automatic 3d hair model.  ACM TOG, 2018.
  * [33] David B Lindell, Julien NP Martel, and Gordon Wetzstein.  Autoint: Automatic integration for fast neural volume rendering.  arXiv preprint arXiv:2012.01714, 2020.
  * [34] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In ECCV, 2018.
  * [35] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  In NeurIPS, 2020.
  * [36] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  NeurIPS, 2020.
  * [37] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  ACM TOG, 2019.
  * [38] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  arXiv:2008.02268, 2020.
  * [39] Abhimitra Meka, Rohit Pandey, Christian Haene, Sergio Orts-Escolano, Peter Barnum, Philip Davidson, Daniel Erickson, Yinda Zhang, Jonathan Taylor, Sofien Bouaziz, Chloe Legendre, Wan-Chun Ma, Ryan Overbeck, Thabo Beeler, Paul Debevec, Shahram Izadi, Christian Theobalt, Christoph Rhemann, and Sean Fanello.  Deep relightable textures - volumetric performance capture with neural rendering.  ACM TOG, 2020.
  * [40] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM TOG, 2019.
  * [41] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  ECCV, 2020.
  * [42] Koki Nagano, Huiwen Luo, Zejian Wang, Jaewoo Seo, Jun Xing, Liwen Hu, Lingyu Wei, and Hao Li.  Deep face normalization.  ACM TOG, 2019.
  * [43] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger.  Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks.  arXiv preprint arXiv:2103.03231, 2021.
  * [44] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In ICCV, 2019.
  * [45] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In CVPR, 2020.
  * [46] Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu.  3D Ken Burns effect from a single image.  ACM TOG, 38(6):1–15, 2019.
  * [47] Yuval Nirkin, Yosi Keller, and Tal Hassner.  FSGAN: Subject agnostic face swapping and reenactment.  In ICCV, 2019.
  * [48] Eric Penner and Li Zhang.  Soft 3d reconstruction for view synthesis.  ACM TOG, 2017.
  * [49] Shengju Qian, Kwan-Yee Lin, Wayne Wu, Yangxiaokang Liu, Quan Wang, Fumin Shen, Chen Qian, and Ran He.  Make a face: Towards arbitrary high fidelity face manipulation.  In ICCV, 2019.
  * [50] Amit Raj, Michael Zollhoefer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, and Stephen Lombardi.  Pva: Pixel-aligned volumetric avatars.  arXiv preprint arXiv:2101.02697, 2021.
  * [51] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  ICLR, 2017.
  * [52] Gernot Riegler and Vladlen Koltun.  Free view synthesis.  ECCV, 2020.
  * [53] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [54] Matan Sela, Elad Richardson, and Ron Kimmel.  Unrestricted facial geometry reconstruction using image-to-image translation.  In CVPR, 2017.
  * [55] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang.  3d photography using context-aware layered depth inpainting.  In CVPR, 2020.
  * [56] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  2019\.
  * [57] Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein.  Metasdf: Meta-learning signed distance functions.  NeurIPS, 2020.
  * [58] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In NeurIPS, 2019.
  * [59] Shih-Yang Su, Frank Yu, Michael Zollhoefer, and Helge Rhodin.  A-nerf: Surface-free human 3d pose refinement via neural rendering.  arXiv preprint arXiv:2102.06199, 2021.
  * [60] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In CVPR, 2019.
  * [61] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng.  Learned initializations for optimizing coordinate-based neural representations.  In CVPR, 2021.
  * [62] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt.  Stylerig: Rigging stylegan for 3d control over portrait images.  In CVPR, 2020.
  * [63] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  ACM TOG, 2019.
  * [64] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  In CVPR, 2016.
  * [65] Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang.  Cross-domain few-shot classification via learned feature-wise transformation.  In ICLR, 2020.
  * [66] Richard Tucker and Noah Snavely.  Single-view view synthesis with multiplane images.  In CVPR, 2020.
  * [67] Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy.  Synthetic depth-of-field with a single-camera mobile phone.  ACM TOG, 2018.
  * [68] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro.  Few-shot video-to-video synthesis.  2019\.
  * [69] Ziyan Wang, Timur Bagautdinov, Stephen Lombardi, Tomas Simon, Jason Saragih, Jessica Hodgins, and Michael Zollhöfer.  Learning compositional radiance fields of dynamic human heads.  arXiv preprint arXiv:2012.09955, 2020.
  * [70] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In CVPR, 2020.
  * [71] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In CVPR, 2020.
  * [72] Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, and Chen Change Loy.  Reenactgan: Learning to reenact faces via boundary transfer.  In ECCV, 2018.
  * [73] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim.  Space-time neural irradiance fields for free-viewpoint video.  arXiv preprint arXiv:2011.12950, 2020.
  * [74] Sicheng Xu, Jiaolong Yang, Dong Chen, Fang Wen, Yu Deng, Yunde Jia, and Xin Tong.  Deep 3d portrait from a single image.  In CVPR, 2020.
  * [75] Xiaogang Xu, Ying-Cong Chen, and Jiaya Jia.  View independent generative adversarial network for novel view synthesis.  In ICCV, 2019.
  * [76] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In CVPR, 2021.
  * [77] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  In ECCV, 2020.
  * [78] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In ICCV, 2019.
  * [79] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv:2010.07492, 2020.
  * [80] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [81] Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, et al.  Neural light transport for relighting and view synthesis.  ACM TOG, 2020.
  * [82] Yajie Zhao, Zeng Huang, Tianye Li, Weikai Chen, Chloe LeGendre, Xinglei Ren, Ari Shapiro, and Hao Li.  Learning perspective undistortion of portraits.  In ICCV, 2019.
  * [83] Xiangyu Zhu, Zhen Lei, Junjie Yan, Dong Yi, and Stan Z Li.  High-fidelity pose and expression normalization for face recognition in the wild.  In CVPR, 2015.


Two Eigenvectors for the price of one

Juan Tolosa

Stockton University

<EMAIL_ADDRESS>
###### Abstract

Starting from a mistake done by a student, we discover an unexpected method of
finding both eigenvectors for a $2\times 2$ matrix with distinct eigenvalues
in a single computation. We discuss a connection with the Cayley-Hamilton
theorem, and show the corresponding generalization for a $3\times 3$ matrix.
The arguments should be understandable for strong linear algebra students.

##  1 Introduction. The $2\times 2$ case.

It all started with a student’s mistake. In a quiz, students were asked to
find eigenvalues and eigenvectors of the matrix

$A=\begin{pmatrix}4&1\cr 2&5\end{pmatrix}$

One student correctly found the two eigenvalues $\lambda_{1}=3$ and
$\lambda_{2}=6$. She also correctly computed the matrix

$B_{1}=A-\lambda_{1}I=\begin{pmatrix}1&1\cr 2&2\cr\end{pmatrix}$

(where $I$ is the identity matrix), only instead of solving the homogeneous
system

$B_{1}{\bf v}_{1}={\bf 0},$

to find the corresponding eigenvector ${\bf v}_{1}={x\choose y}$ for
$\lambda_{1}=3$, she picked a column of $B_{1}$ and (wrongly) declared that
this was the required eigenvector, instead of picking, say, ${\bf
v}_{1}={-1\choose 1}$.

To my surprise, however, I realized that the chosen vector, ${\bf
v}_{2}={1\choose 2}$, is actually an eigenvector for the second eigenvalue
$\lambda_{2}=6$!

At first I decided this might be just a coincidence. Upon checking the general
case of a real-valued $2\times 2$ matrix $A$ with distinct eigenvalues, this
turned out to be true in general. The direct check could be a good exercise
for a motivated student.

We can, instead, seek a deeper understanding of the situation and, in doing
so, discover a connection with an important result in linear algebra. Namely,
assuming $A$ has distinct eigenvalues $\lambda_{1}\neq\lambda_{2}$, if we call
$B_{2}=A-\lambda_{2}I$ the matrix we use to find the second eigenvector, then
the fact that the columns of $B_{1}$ turn out to be solutions ${\bf v}_{2}$ of
$B_{2}{\bf v}_{2}={\bf 0}$ is equivalent to saying that the matrix product
$B_{2}B_{1}$ is equal to the zero matrix. But if we think a little, we realize
that

$B_{2}B_{1}=(A-\lambda_{2}I)(A-\lambda_{1}I)=A^{2}-(\lambda_{1}+\lambda_{2})A+\lambda_{1}\lambda_{2}I$

is equal to $p(A)$, where $p(\lambda)$ is the characteristic polynomial of
$A$, $p(\lambda)=\lambda^{2}-({\rm trace}\,A)\lambda+{\rm(determinant\,}A)$.
And the fact that $p(A)$ is indeed zero is the celebrated Cayley-Hamilton
theorem, stating that every square matrix satisfies its own characteristic
equation; see, for example, [1] or [2].

Thus, for a $2\times 2$ matrix $A$ with distinct eigenvalues, the computations
to find the first eigenvector $\lambda_{1}$ will also provide the second
eigenvector for free! We just need to pick any nonzero column vector of
$B_{1}=A-\lambda_{1}I$. Notice that, since we assume eigenvalues are distinct,
then the kernel of $B_{1}$ (the corresponding eigenspace) has dimension 1, and
the rank-nullity theorem guarantees the dimension of the column space of
$B_{1}$ is also $1$, so $B_{1}$ is guaranteed to have at least one nonzero
column.

If $A$ has a double eigenvalue $\lambda_{1}=\lambda_{2}$, then by Cayley-
Hamilton $p(A)=B_{1}^{2}=0$, and we have two possible cases:

  1. 1.

$B_{1}=0$ or, equivalently, $A$ is a multiple of the identity. Here all
nonzero vectors are eigenvectors; we can pick any two linearly independent
vectors to get the basis of the eigenspace.

  2. 2.

$B_{1}\neq 0$, in which case the eigenspace has dimension one; any nonzero
column of $B_{1}$ will provide an eigenvector. If we want a basis in which $A$
has canonical (Jordan) form, then, besides an eigenvector, we need to also
pick a generalized eigenvector (see [1], p. 66). Here is how we can get both.
Since $B_{1}^{2}=0$, one needs just pick any vector ${\bf w}$ not in the
kernel of $B_{1}$ (that is, any nonzero vector which is not an eigenvector),
and compute ${\bf v}=B_{1}{\bf w}$. Then ${\bf v}$ will be an eigenvector of
$A$, with generalized eigenvector ${\bf w}$. Namely, we will have

$A{\bf v}=\lambda{\bf v}\qquad\hbox{ and }\qquad A{\bf w}=\lambda{\bf w}+{\bf
v}.$

For example, for the matrix

$A=\begin{pmatrix}2&1\cr-1&4\end{pmatrix},$

with double eigenvalue $\lambda=3$, we have

$B=A-3I=\begin{pmatrix}-1&1\cr-1&1\end{pmatrix}.$

if we pick ${\bf w}=(1,0)$, we get the eigenvector ${\bf v}=B{\bf w}=(-1,-1)$,
with generalized eigenvector ${\bf w}$.

##  2 The $3\times 3$ case.

What about a $3\times 3$ matrix $A$? First let us assume that $A$ has three
distinct eigenvalues $\lambda_{1},\lambda_{2},\lambda_{3}$, and compute the
corresponding matrices $B_{i}=A-\lambda_{i}I$, $i=1,2,3$.

Then the Cayley-Hamilton theorem will imply that $B_{1}B_{2}B_{3}=0$, so the
nonzero columns of the product $B_{2}B_{3}$ will be eigenvectors of
$\lambda_{1}$. For example, the matrix

$A=\begin{pmatrix}7&-4&-5\cr 3&-2&-3\cr 6&-4&-4\end{pmatrix}$

has eigenvalues $\lambda_{1}=1$, $\lambda_{2}=2$, and $\lambda_{3}=-2$. Here

$B_{2}=\begin{pmatrix}5&-4&-5\cr 3&-4&-3\cr 6&-4&-6\end{pmatrix},\qquad
B_{3}=\begin{pmatrix}9&-4&-5\cr 3&0&-3\cr 6&-4&-2\end{pmatrix},$

and

$B_{2}B_{3}=\begin{pmatrix}3&0&-3\cr-3&0&3\cr 6&0&-6\end{pmatrix}.$

The first and the last columns provide an eigenvector for $\lambda_{1}=1$,
which can be simplified to $(1,-1,2)$.

Notice that $B_{2}B_{3}$ must have at least one nonzero column. Indeed, if the
three eigenvalues are distinct, then $p(\lambda)$ is the minimal polynomial of
$A$ ([1], p. 179), whereas $(\lambda-\lambda_{2})(\lambda-\lambda_{3})$ has
smaller degree, so $B_{2}B_{3}=(A-\lambda_{2}I)(A-\lambda_{3}I)$ cannot be the
zero matrix.

Assume now $A$ has a simple eigenvalue $\lambda_{1}$ and a double eigenvalue
$\lambda_{2}$. Then we have two sub-cases.

  1. 1.

The eigenspace of $\lambda_{2}$ has dimension two. Then the minimal polynomial
of $A$ is $(\lambda-\lambda_{1})(\lambda-\lambda_{2})$, so that not only
$B_{2}^{2}B_{1}=0$, but also $B_{2}B_{1}=0$; therefore, the nonzero columns of
$B_{1}$ are eigenvectors of the eigenvalue $\lambda_{2}$. Moreover, since
$\lambda_{1}$ is a simple eigenvalue, the kernel of $B_{1}$ has dimension $1$;
hence, the column space of $B_{1}$ has dimension $2$. In other words, the
column space of $B_{1}$ generates the entire (two-dimensional) eigenspace. So
here you get three eigenvectors for the price of one! As an example, consider
the matrix

$A=\begin{pmatrix}4&-9&-6\cr-6&7&6\cr 12&-18&-14\end{pmatrix}$

with $\lambda_{1}=1$ and $\lambda_{2}=\lambda_{3}=-2$. Here

$B_{1}=\begin{pmatrix}3&-9&-6\cr-6&6&6\cr 12&-18&-15\end{pmatrix}$

can be used to solve $B_{1}{\bf v}={\bf 0}$ and find the eigenvector ${\bf
v}_{1}=(1,-1,2)$, corresponding to $\lambda_{1}=1$. Moreover, a direct check
shows that all columns of $B_{1}$ are eigenvectors for
$\lambda_{2}=\lambda_{3}=-2$. Since $\dim(\mathop{\rm col}\nolimits(B_{1}))=2$
(because $\lambda_{1}$ is a simple eigenvalue), we conclude that the
eigenspace of $\lambda_{2}$ is two-dimensional, and the columns of $B_{1}$
generate this eigenspace. For example, the first two columns of $B$ (suitably
divided by $3$) provide the missing eigenvectors ${\bf v}_{2}=(1,-2,4)$ and
${\bf v}_{3}=(-3,2,-6)$ for $\lambda_{2}$.

  2. 2.

The eigenspace of $\lambda_{2}$ has dimension one. Then the characteristic
polynomial $p(\lambda)=(\lambda-\lambda_{1})(\lambda-\lambda_{2})^{2}$ is also
the minimal polynomial. Hence, $B_{2}^{2}B_{1}=0$ but $B_{2}B_{1}\neq 0$. As
in the first case, the column space of $B_{1}$ has dimension 2. Moreover, in
this case the kernel of $B_{2}$ has dimension 1, and the kernel of $B_{2}^{2}$
has dimension 2. (It has dimension at least 2, since $\mathop{\rm
col}\nolimits(B_{1})\subset\ker(B_{2}^{2})$, and cannot have dimension $3$,
since otherwise $A$ would be a multiple of the identity, and $\lambda_{2}$
would be a triple eigenvalue.) Consequently, the column space of $B_{1}$
generates the entire set of eigenvectors and generalized eigenvectors of
$\lambda_{2}$. Hence, to find an eigenvector and generalized eigenvector for
$\lambda_{2}$ we may proceed as follows:

     * •

Pick a nonzero column ${\bf w}$ of $B_{1}$ that is not an eigenvector of
$B_{2}$, that is, such that $B_{2}{\bf w}$ is nonzero. Such column must exist,
since $\dim(\mathop{\rm col}\nolimits(B_{1}))=2$ and $\dim(\ker(B_{2}))=1$.

     * •

Compute ${\bf v}=B_{2}{\bf w}$. Then $B_{2}{\bf v}=B_{2}^{2}{\bf w}=0$, since
${\bf w}\in\mathop{\rm col}\nolimits(B_{1})=\ker(B_{2}^{2})$. We conclude that
${\bf v}$ is an eigenvector, and ${\bf w}$ is the corresponding generalized
eigenvector, for $\lambda_{2}$.

As an example, consider

$A=\begin{pmatrix}5&-10&-7\cr-6&7&6\cr 13&-19&-15\end{pmatrix},$

with eigenvalues $\lambda_{1}=1$ and $\lambda_{2}=\lambda_{3}=-2$. Here

$B_{1}=A-\lambda_{1}I=\begin{pmatrix}4&-10&-7\cr-6&6&6\cr
13&-19&-16\end{pmatrix}.$

As usual, we use $B_{1}$ to find an eigenvector for $\lambda_{1}$, say, ${\bf
v}_{1}=(1,-1,2)$.

Next, a direct check shows that not all columns of $B_{1}$ are eigenvectors
for $\lambda_{2}$ (in fact, neither column of $B_{1}$ is). This tells us that
the eigenspace for $\lambda_{2}$ has dimension 1 (since otherwise all columns
of $B_{1}$ would be eigenvectors). To proceed, we can therefore pick any
nonzero column of $B_{1}$, say, the first column:

${\bf w}=\begin{pmatrix}4\\\ -6\\\ 13\end{pmatrix},$

and calculate

${\bf v}_{2}=B_{2}{\bf w}=\begin{pmatrix}-3\\\ 0\\\ -3\end{pmatrix}.$

We conclude that ${\bf v}_{2}$ is an eigenvector for $\lambda_{2}$, with
corresponding generalized eigenvector ${\bf w}$. In the basis $\\{{\bf
v}_{1},{\bf v}_{2},{\bf w}\\}$, the matrix $A$ will have (Jordan) canonical
form.

##  3 A triple eigenvalue.

If $A$ is a $3\times 3$-matrix with a triple eigenvalue $\lambda$, we have
three possibilities (Let us again denote $B=A-\lambda I$).

  1. 1.

The eigenspace of $\lambda$ has dimension three, then $B=0$, and all nonzero
vectors are eigenvectors.

  2. 2.

The eigenspace of $\lambda$ has dimension one. Then $B^{2}\neq 0$ and has rank
one; any nonzero column of $B^{2}$ will provide an eigenvector. Moreover, if
we want a basis in which $A$ will have canonical form, we can get two
generalized vectors as follows: Pick any nonzero vector ${\bf v}_{1}$ that is
not in the kernel of $B$; compute ${\bf v}_{2}=B{\bf v}_{1}$, and ${\bf
v}_{3}=B{\bf v}_{2}$. Then ${\bf v}_{3}$ is an eigenvector of $A$, with two
generalized eigenvectors ${\bf v}_{2}$ and ${\bf v}_{1}$.

  3. 3.

The eigenspace of $\lambda$ has dimension two. Then $B\neq 0$, $B^{2}=0$, and
$B$ has rank one. Any nonzero column of $B$ will be an eigenvector of $A$.
Moreover, if we choose any vector ${\bf v}$ not in the kernel of $B$, and
compute ${\bf w}=B{\bf v}$, we will have both an eigenvector ${\bf w}$ and a
corresponding generalized vector ${\bf v}$. Unfortunately, here we will still
be missing a second (independent) eigenvector. As it turns out, however, a
careful analysis of the linear dependence of the columns of $B$ does provide
two independent eigenvectors; see Section 5.

##  4 What if we don’t want to solve any linear systems?

It turns out in all cases we can find the eigenvectors without solving any
linear systems! Here is how we do it (the missing justifications are in
previous sections).

  * •

$2\times 2$-matrix $A$.

    1. 1.

If the eigenvalues $\lambda_{1},\lambda_{2}$ are distinct, compute
$B_{i}=A-\lambda_{i}I$; any nonzero column of $B_{1}$ is an eigenvector for
$\lambda_{2}$, and vice-versa.

    2. 2.

If $\lambda_{1}=\lambda_{2}$ and the eigenspace has dimension two, then $A$ is
a multiple of the identity; pick any two linearly independent vectors as a
basis for the eigenspace.

    3. 3.

If $\lambda_{1}=\lambda_{2}$ and the eigenspace has dimension one, then any
nonzero column of $B_{1}$ will generate the eigenspace.

  * •

$3\times 3$-matrix $A$.

    1. 1.

If eigenvalues $\lambda_{i}$ are distinct, compute $B_{i}=A-\lambda_{i}I$,
$i=1,2,3$. Then any nonzero column of $B_{i}B_{j}$ is an eigenvector for the
third $\lambda_{k}$.

    2. 2.

If $\lambda_{1}\neq\lambda_{2}=\lambda_{3}$. Since $B_{1}B_{2}^{2}=0$, then
any nonzero column of $B_{2}^{2}$ spans the eigenspace for $\lambda_{1}$.
Moreover,

       * –

If $B_{2}B_{1}=0$, then the eigenspace of $\lambda_{2}$ has dimension two, and
is spanned by any two linearly independent columns of $B_{1}$.

       * –

If $B_{2}B_{1}\neq 0$, then the eigenspace of $\lambda_{2}$ is one-
dimensional, and is spanned by any nonzero column of $B_{2}B_{1}$.

    3. 3.

If $\lambda$ is a triple eigenvalue, we have three possibilities.

       * –

The eigenspace of $\lambda$ has dimension three. Then $A$ is a multiple of the
identity, and any (nonzero) vector is an eigenvector.

       * –

The eigenspace of $\lambda$ has dimension one. Then $B^{2}\neq 0$, and any
nonzero column of $B^{2}$ will span this eigenspace.

       * –

The eigenspace of $\lambda$ has dimension two. Then $B\neq 0$, $B^{2}=0$, and
$B$ has rank one. Any nonzero column ${\bf v}$ of $B$ will provide an
eigenvector, but we are still missing a second eigenvector. It appears at
first sight that the columns of $B$ provide no means of finding the entire
eigenspace of $\lambda$ without solving a linear system. In the following
Section we will show, however, that this is not so.

##  5 More on the special case

In Section 4 we showed how to find the eigenspace of a $3\times 3$ matrix $A$
using the columns of $B_{i}=A-\lambda_{i}I$ in all cases but one, Case 3, when
$A$ has a triple eigenvalue $\lambda$ with a two-dimensional eigenspace. Then
the column space of $B=A-\lambda I$ is one dimensional, and provides one of
the two independent eigenvectors of the eigenspace of $\lambda$; the
corresponding eigenvalue for $B$ is zero. The following result is
straightforward.

A matrix $B$ has a triple eigenvalue zero, with two-dimensional eigenspace if,
and only if, $B\neq 0$ and $B^{2}=0$.

For such a matrix the column space is one-dimensional, whereas the eigenspace
is two-dimensional. Any nonzero column of $B$ provides an eigenvector, but we
are missing a second one.

It turns out, however, that the columns of $B$ do provide the two required
eigenvectors; it is just a matter of analyzing the specific linear dependence
of the columns of $B$, as we will see below.

The proofs of all the results stated in this section are skipped; they can be
carried out by direct calculation.

Since the column space of $B$ is one-dimensional, it follows that all columns
of $B$ are proportional to one of them (any nonzero column).

However, this condition is not sufficient to guarantee that $B^{2}$ is the
zero matrix, as seen by the example

$B=\begin{pmatrix}1&1&1\cr 1&1&1\cr 1&1&1\end{pmatrix}.$

To figure out sufficient conditions, and the structure of the eigenspace, let
us divide the discussion into three cases.

###  5.1 Case 1. The first column of $B$ is nonzero.

Then the matrix $B$ has the following structure, by columns:

$B=\left({\bf v}\,\left|t{\bf v}\right|s{\bf v}\right),$ (1)

where

${\bf v}=\begin{pmatrix}x\cr y\cr z\end{pmatrix}$

is the first column of $B$, which we assume to be nonzero, and $t,s$ are
arbitrary real numbers.

Here we can prove the following fact.

The matrix $B$ satisfies $B^{2}=0$ if, and only if,

$x+ty+sz=0.$

Moreover, in this case, the eigenspace of the (triple) zero eigenvalue is
spanned by the vectors

${\bf v}_{1}=\begin{pmatrix}-t\cr 1\cr 0\end{pmatrix}\qquad\hbox{ and
}\qquad{\bf v}_{2}=\begin{pmatrix}-s\cr 0\cr 1\end{pmatrix}.$ (2)

Thus, in Case 1, the matrix $B$ has the form (1), with

${\bf v}=\begin{pmatrix}-ty-sz\cr y\cr z\end{pmatrix},$

and the eigenspace of zero is spanned by the vectors (2).

Such matrix $B$ constitutes a four-parameter family of matrices, with
arbitrary parameters $t,s,y,z$, with the only condition that not both $(y,z)$
are zero.

Example. An example with triple eigenvalue $3$:

$A=\begin{pmatrix}-2&5&-10\cr-1&4&-2\cr 2&-2&7\end{pmatrix};\qquad
B=A-3I=\begin{pmatrix}-5&5&-10\cr-1&1&-2\cr 2&-2&4\end{pmatrix},$

so here $t=-1$ and $s=2$. It follows that the eigenspace of the eigenvalue $3$
is spanned by ${\bf v}_{1}=\begin{pmatrix}1\cr 1\cr 0\end{pmatrix}$ and ${\bf
v}_{2}=\begin{pmatrix}-2\cr 0\cr 1\end{pmatrix}.$ Of course, the columns of
$B$ also belong to this eigenspace.

Moreover, if needed, it is not hard to indicate a generalized eigenvector for
a general matrix in Case 1. Since the first column ${\bf v}$ is an
eigenvector, and since $B{\bf e}_{1}=$ the first column of $B$, where ${\bf
e}_{1}=\begin{pmatrix}1\cr 0\cr 0\end{pmatrix}$, it follows that ${\bf e}_{1}$
is a generalized eigenvector, corresponding to the eigenvector ${\bf v}$.

###  5.2 Case 2. The first column of $B$ is zero, the second column is
nonzero.

Then the matrix $B$ has the following form, by columns:

$B=\left({\bf 0}\,\left|{\bf v}\right|t{\bf v}\right),$ (3)

where

${\bf v}=\begin{pmatrix}x\cr y\cr z\end{pmatrix}$

is the second (nonzero) column of $B$, and $t$ is an arbitrary real number.

The following result holds in this case:

The matrix (3) satisfies $B^{2}=0$ if, and only if,

$y+tz=0.$

Moreover, in this case, the eigenspace of the (triple) zero eigenvalue is
spanned by the vectors

${\bf v}_{1}=\begin{pmatrix}0\cr-t\cr 1\end{pmatrix}\qquad\hbox{ and
}\qquad{\bf v}_{2}={\bf e}_{1}=\begin{pmatrix}1\cr 0\cr 0\end{pmatrix}.$ (4)

It follows that in Case 2 the matrix $B$ has the form (3), with

${\bf v}=\begin{pmatrix}x\cr-tz\cr z\end{pmatrix},$

and the eigenspace of zero is spanned by the vectors (4).

Here we have a three-parameter family of matrices $B$, with parameters
$t,x,z$, with the only condition that $(x,z)$ are not simultaneously zero.

Example. A matrix with triple eigenvalue $2$:

$A=\begin{pmatrix}2&-1&2\cr 0&-4&12\cr 0&-3&8\end{pmatrix};\qquad
B=A-2I=\begin{pmatrix}0&-1&2\cr 0&-6&12\cr 0&-3&6\end{pmatrix},$

so $t=-2$, and therefore the eigenspace of $\lambda=2$ is spanned by ${\bf
v}_{1}=\begin{pmatrix}0\cr 2\cr 1\end{pmatrix}$ and ${\bf
e}_{1}=\begin{pmatrix}1\cr 0\cr 0\end{pmatrix}.$ Moreover, ${\bf
e}_{2}=\begin{pmatrix}0\cr 1\cr 0\end{pmatrix}$ is a generalized vector
corresponding to the eigenvector ${\bf
v}=\begin{pmatrix}-1\cr-6\cr-3\end{pmatrix}$, since $B{\bf e}_{2}={\bf v}=$
second column of $B$.

###  5.3 Case 3. The first two columns of $B$ are zero, the last column is
nonzero.

Here the following holds.

In Case 3, the matrix $B$ satisfies $B^{2}=0$ if, and only if, it has the form

$B=\begin{pmatrix}0&0&x\cr 0&0&y\cr 0&0&0\end{pmatrix},$

where $x,y$ are arbitrary numbers, not both zero. Moreover, in this case, the
eigenspace of the (triple) zero eigenvalue is spanned by the vectors

${\bf e}_{1}=\begin{pmatrix}1\cr 0\cr 0\end{pmatrix}\qquad\hbox{ and
}\qquad{\bf e}_{2}=\begin{pmatrix}0\cr 1\cr 0\end{pmatrix}.$ (5)

Finally, ${\bf e}_{3}=\begin{pmatrix}0\cr 0\cr 1\end{pmatrix}$ is a
generalized vector for the eigenvector ${\bf v}=\begin{pmatrix}x\cr y\cr
0\end{pmatrix}.$

##  6 Acknowledgment.

I am grateful to the editors of the College Mathematics Journal for most of
the content of Section 4.

## References

  * [1] Axler, S. (1997). Linear Algebra Done Right. New York: Springer; p. 173 and p. 207.
  * [2] Crilly, T. (1992). A Gemstone in Matrix Algebra Math. Gazette 76(475): 182–188. doi.org/10.2307/3620391.