# Progressive Learning of 3D Reconstruction Network from 2D GAN Data

Aysegul Dundar, Jun Gao, Andrew Tao, Bryan Catanzaro  A. Dundar, J. Gao, A.
Tao, B. Catanzaro are with NVIDIA, CA, USA. A. Dundar is with Department of
Computer Science, Bilkent University, Ankara, Turkey J. Gao is with Department
of Computer Science, University of Toronto, Canada.

###### Abstract

This paper presents a method to reconstruct high-quality textured 3D models
from single images. Current methods rely on datasets with expensive
annotations; multi-view images and their camera parameters. Our method relies
on GAN generated multi-view image datasets which have a negligible annotation
cost. However, they are not strictly multi-view consistent and sometimes GANs
output distorted images. This results in degraded reconstruction qualities. In
this work, to overcome these limitations of generated datasets, we have two
main contributions which lead us to achieve state-of-the-art results on
challenging objects: 1) A robust multi-stage learning scheme that gradually
relies more on the models own predictions when calculating losses, 2) A novel
adversarial learning pipeline with online pseudo-ground truth generations to
achieve fine details. Our work provides a bridge from 2D supervisions of GAN
models to 3D reconstruction models and removes the expensive annotation
efforts. We show significant improvements over previous methods whether they
were trained on GAN generated multi-view images or on real images with
expensive annotations. Please visit our web-page for 3D visuals:
https://research.nvidia.com/labs/adlr/progressive-3d-learning.

###### Index Terms:

3D Texture Learning, 3D Reconstruction, Single-image Inference, Generative
Adversarial Networks.

##  1 Introduction

GAN based models achieve realistic image synthesis on various objects [19, 20,
55] and find applications in image editing, conditional image generation [39,
9, 27], and video generation [49] tasks. They are also found to be useful for
dataset generations with automatic part segmentation annotations [59, 44].
There is further interest to deploy this technology for gaming, robotics,
architectural designs, and AR/VR applications. However, such applications also
require contollability on the viewpoint requiring generation in 3D
representations. On the other hand, the realism of 3D image generation and
reconstruction results are not on par with the GAN generated 2D images [2, 36,
53, 3, 10]. In this work, we are interested in closing this gap.

To reconstruct high quality 3D models, current state-of-the-art (SOTA) methods
rely on 3D annotations. Such data is expensive to collect, requires special
hardware and is usually collected in constrained lab environments. Due to the
difficulty in collecting such annotations, efforts have been limited to few
objects such as faces [11] and human bodies [56, 23]. A cheaper alternative is
finely curated multi-view datasets. They can be collected with a camera
without expensive hardware requirements. However, they are still difficult to
annotate for their camera parameters. For that reason, synthetic images are
used instead of real images to train 3D reconstruction models [5, 7, 6, 53].
While these models learn to reconstruct synthetic objects, they fall short in
their ability to recover the 3D properties of real images due to the domain
gap between synthetic and real images.

Single view image collections are also explored to learn 3D reconstruction
models [17, 2]. However, with single view images during trainings, models
receive limited supervision; only to the visible parts. Various constraints
and regularizers are proposed to obtain plausible results such as losses which
limit the deformation from mean template [17, 6], rotation and swap
consistency losses [2, 36] and semantic consistency [24] constraints. Still
results are not realistic. Another way to use single view image collections to
learn 3D reconstruction models is to train a GAN model from them and generate
multi-view datasets with the trained GAN models [58, 26]. This approach
becomes possible because recent generative models of images, especially
StyleGANs, are shown to learn an implicit 3D representation with latent codes
that can be manipulated to change the viewpoint of a scene [19]. The latent
codes of the StyleGAN are controlled to generate consistent objects from
different view points. Few selected viewpoints are labeled for camera
parameters which only takes a minute to annotate and unlimited number of
samples can be generated [58] for those viewpoints. However, one issue of
these datasets is that they are not perfect especially in presenting the
realistic details across views. This is because StyleGAN does not have strict
disentanglement of shape, texture, and camera parameters. Therefore, one
cannot change the camera parameters while preserving the identity strictly.
Another issue is the distorted image generations, sometimes appear as missing
parts in objects (cf. Fig. 2).

Figure 1: Given a single 2D image input, our method outputs high quality
textured 3D models. We achieve these results by learning from StyleGAN
generated datasets via a robust multi-stage training scheme and a novel
adversarial learning pipeline.

In this work, our goal is to learn accurate 3D reconstruction models from GAN
generated multi-view images. As our first contribution, we propose a framework
that is robust to the noise in the training data. We achieve this with a
multi-stage learning scheme that gradually relies more on the models own
predictions when calculating losses. Secondly, we propose a novel adversarial
learning pipeline with online pseudo-ground truth generation to train a
discriminator. With the discriminator, our model learns to output fine
details. Our model shows significant improvements over previous methods
whether they were trained on GAN generated multi-view images or on real images
with expensive data collections/annotations pipelines. In summary, our main
contributions are:

  * •

A robust multi-stage learning scheme that relies more on the model’s
predictions at each step. Our model is not affected by missing parts in the
images and inconsistencies across views.

  * •

A novel adversarial learning pipeline to increase the realism of textured 3D
predictions. We generate pseudo-ground truth during training and employ a
multi-view conditional discriminator for learning to generate fine details.

  * •

High-fidelity textured 3D model synthesis both qualitatively and
quantitatively on three challenging objects. Examples are shown in Fig. 1.

##  2 Related Work

Style-based GAN models [19, 20] achieve high quality synthesis of various
objects which are quite indistinguishable from real image and are shown to
learn an implicit 3D knowledge of objects without a supervision. One can
control the viewpoint of the synthesized object by its latent codes. This
makes pretrained GANs a promising technology for controllable generation [48,
1, 41]. However, in these models, the disentanglement of 3D shape and
appearance is not strict and therefore the appearance of objects change as the
viewpoint is manipulated. Recently, 3D-aware generative models are proposed
with impressive results but they either do not guarantee strict 3D consistency
[37, 38, 4, 13] or computationally expensive [3] and overall not on par with
2D StyleGAN results [10]. We are interested in single-view image inference so
our work is more related to image inversion methods that project images into
these 3D-aware GAN’s latent space [52, 22, 29]. Even though significant
progress is achieved for image inversion [52, 22, 29], these methods require
run-time optimization and suffer from lower quality novel view predictions.

Figure 2: Overview of the dataset generation (a) and multi-stage training
scheme of the reconstruction network (b). The generator network takes input
image and outputs mesh and texture predictions. In the first stage, the output
is rendered from another view than the input image and losses are calculated
on this novel view. This way model is not effected by the missing parts in the
images and also the unrealistic segmentation maps resulted from these images.
In the second stage, additional reconstruction loss is added from the same
view. The rendered and ground-truth images are masked based on the silhouette
predictions of the model. Lastly, to achieve sharp and realistic predictions,
we add adversarial training on the third stage. GAN training pipeline is given
in Fig. 3.

There have been many works that learn textured 3D mesh models from images with
differentiable renderers [30, 21, 28, 6, 43]. Deep neural networks are coupled
with the renderers and trained to predict 3D mesh representations and texture
maps of input images via reconstruction losses [17, 6, 12, 24, 15]. However,
inferring these 3D attributes from single 2D images is inherently ill-posed
problem given that the invisible mesh and texture predictions receive no
gradients during training [17, 6, 12, 24, 15]. These algorithms that learn
from single-view images output results that look unrealistic especially when
viewed from a different point.

Multi-view image datasets provide a solution for the limited supervision
problem of single-view image datasets. However, due to the expensive
annotation of multi-view image datasets for their 2D keypoints or camera pose,
they are small in scale. There have been methods that use sequence of images
to optimize a mesh and texture model [8]. However, they learn a new network
for each sequence. Recently, these sequence of image datasets have also been
tremendously explored with a method called Neural Radiance Fields (Nerf) to
explore implicit geometry [34, 31, 50]. These models overfit to a sequence and
can not be used for single image inference. PixelNerf [53] is an extension of
these models that achieves single image inference, however, as we show in our
experiments, the results are not good.

Another promising direction with Nerf-based models is optimization of 3D
representations with well-trained diffusion models. These models can stylize
meshes or generate 3D geometry representations from scratch with given text
prompts [32, 25, 33, 42]. However, these models require run time-optimization
and control on the generation is limited. In our work, we are interested in a
different application, single view image reconstruction where the generation
is conditioned on an input image.

In our work, we are interested in mesh representations due its efficiency in
rendering. To infer mesh representations, multi-view datasets are also shown
to be beneficial based on the experiments with synthetic datasets [5, 46, 45].
However, those results do not translate well to real image inferences because
of the domain gap between synthetic and real images. To generate a realistic
multi-view dataset with cheap labor cost, Zhang et. al. [58] use a generative
adversarial network by controlling the latent codes and generate coarsely
consistent objects from different view points. We also use these datasets but
achieve significantly better results than the state-of-the-art and Zhang et.
al. [58] thanks to the robust learning scheme (which also allows us to remove
regularizers that limit the deformations) and the adversarial learning
pipeline.

##  3 Method

In Section 3.1, we describe the motivation of our approach. The multi-stage
training scheme and adversarial learning pipeline are presented in Section 3.2
and 3.3, respectively.

###  3.1 Motivation

Differentiable rendering enable training neural networks to perform 3D
inference such as predicting 3D mesh geometry and textures from images [6].
However, they require multi-view images, camera parameters, and object
silhouettes to achieve high performance models. Such data is expensive to
obtain. StyleGAN generated datasets remove the expensive labeling effort via
the latent codes that control the camera viewpoints. When few viewpoints are
selected and annotated, multi-view images can be generated in infinite numbers
for those viewpoints. The annotations require 1 minute [58] no matter how many
images are generated because they are all aligned across different examples.
As for the segmentation masks the renderers utilize during training, they can
be obtained by off-the-shelf instance segmentation models [14]. However,
learning a high performing model from these datasets remains a challenge since
generated images suffer from precise multi-view consistency as shown in Fig. 2
by red rectangles. Additionally, some examples have missing parts as shown in
Fig. 2 by blue rectangles, the head of the horse is not generated in good
quality which also transfers to instance segmentation mask (fourth-row). In
this work, we address these challenges by proposing a robust multi-stage
training pipeline and an adversarial learning set-up.

Figure 3: Gan training pipeline. Discriminator is trained with fake and real
pairs and a conditioning pair. First, texture and mesh predictions are
outputted by the generator. With the estimated mesh and given camera
parameters of a second view image, texture and visibility maps are projected
from the second view image. For fake pair, estimated texture is partially
erased by the visibility map. Additionally, conditioning input pair is
obtained by projecting from the first view image with the estimated mesh
predictions and given camera parameters of the first view.

###  3.2 Multi-stage Training Pipeline

We train the generator that outputs mesh and texture predictions with multi-
stage training pipeline to be robust to the errors in annotations and multi-
view inconsistencies. At each stage of our pipeline, results improve
progressively.

First Stage. In the first stage, our model outputs 3D mesh and texture
predictions from an input image that is from camera view-1, $I^{g}_{v1}$. We
render the image from these 3D predictions with the target view, view-2, and
output the image $I^{r}_{v2}$. We calculate the losses on these novel
predictions. The target view is randomly selected among the sequence. The
motivation of the first stage is to capture reliable 3D mesh predictions and
reasonable texture estimations. At this stage, we do not expect a high quality
texture estimation given the inconsistencies across views. Our experiments
show that when the network is guided with the losses from the same view as
input ($I^{r}_{v1}$ vs. $I^{g}_{v1}$), the errors in missing parts and so the
errors in segmentation mask annotations propagate to the mesh predictions.
This instabilizes the training even when the model is trained with multi-view
consistency, e.g. objectives calculated from both $I^{r}_{v1}$ and
$I^{r}_{v2}$. Therefore, in the first stage, we learn to reconstruct an object
from the image of the object from a different view. This way model does not
overfit to the errors of the given view since it receives feedback from a
novel view. Note that novel view may and does also have errors but since a
novel view is randomly sampled from a sequence and errors are not consistent
among the views, the model outputs the most plausible 3D model to minimize
losses in a sense similar to majority voting.

The training losses at this stage is calculated as follows. We use perceptual
image reconstruction loss between the ground-truth image of the target view
($I^{g}_{v2}$) and the rendered image view ($I^{r}_{v2}$). We mask the images
with ground-truth silhouette (mask) predictions, $S^{g}_{v2}$. This way
reconstruction loss is only calculated on the object. As a reconstruction
loss, we use perceptual losses from Alexnet ($\Phi$) at different feature
layers ($j$) between these images from the loss objective as given in Eq. 1.

$\small\mathcal{L}_{p-nv}=||\Phi_{j}(I^{g}_{v2}*S^{g}_{v2})-\Phi_{j}(I^{r}_{v2}*S^{g}_{v2})||_{2}$
(1)

For shapes, we use an IoU loss between the silhouette rendered ($S^{r}_{v2}$)
and the silhouette ($S^{g}_{v2}$) of the input object.

$\vspace{-1pt}\small\mathcal{L}_{sil}=1-\frac{||S^{g}_{v2}\odot
S^{r}_{v2}||_{1}}{||S^{g}_{v2}+S^{r}_{v2}-S^{g}_{v2}\odot S^{r}_{v2}||}$ (2)

Similar to [6, 28], we also regularize predicted mesh using a laplacian loss
($\mathcal{L}_{lap}$) constraining neighboring mesh triangles to have similar
normals. Following are our base losses:

$\begin{multlined}\small\mathcal{L}_{first}=\lambda_{pn}\mathcal{L}_{p-nv}+\lambda_{s}\mathcal{L}_{sil}+\lambda_{lap}\mathcal{L}_{lap}\end{multlined}\small\mathcal{L}_{first}=\lambda_{pn}\mathcal{L}_{p-nv}+\lambda_{s}\mathcal{L}_{sil}+\lambda_{lap}\mathcal{L}_{lap}$
(3)

The model outputs reliable 3D mesh predictions since it gets feedback from
different views. Note that, we do not use many regularizes such as mean
template and penalizing deformation vertices as previous works [6, 2, 58] and
still achieve a stable training with the first stage objectives.

Second Stage. In the second stage, we rely on our 3D mesh predictions and
introduce additional losses between $I^{r}_{v1}$ vs. $I^{g}_{v1}$, same view
as the input image. In 3D inference predictions, we expect the model to output
predictions that faithfully match with object for the input view. Therefore,
in the second stage, we add additional reconstruction losses from the input
view. However, we do not add silhouette loss for the input view because there
are noise in the segmentation masks. Furthermore, for the reconstruction loss,
we do not mask the input image and rendered output with the ground-truth mask
since it is noisy. We rely on the 3D prediction of our model and mask the
reconstruction loss based on the projected mesh prediction.

The rendered and ground-truth images are masked based on the silhouette
predictions of the model and perceptual loss is calculated as given in Eq. 4.
This way, we rely on the first stage training for the mesh prediction and
learn improved textures with the same view training for the visible parts.
Mesh predictions can still improve via the reconstruction losses since they
still receive a feedback via image reconstruction, but we do not guide it
directly with the silhouettes.

$\small\mathcal{L}_{p-sv}=||\Phi_{j}(I^{g}_{v1}*S^{r}_{v1})-\Phi_{j}(I^{r}_{v1}*S^{r}_{v1})||$
(4)

We use the following loss in additional iterations:

$\begin{multlined}\small\mathcal{L}_{second}=\mathcal{L}_{first}+\lambda_{ps}\mathcal{L}_{p-sv}\end{multlined}\small\mathcal{L}_{second}=\mathcal{L}_{first}+\lambda_{ps}\mathcal{L}_{p-sv}$
(5)

The second stage training starts after first stage training converges. That is
because we rely on the models mesh predictions in the newly introduces losses.
Results improve significantly but the results are not sharp as the training
data.

Third Stage. After learning reliable mesh representation and high quality
texture predictions, we use generative learning pipeline to improve realism of
our predictions. In this stage, we rely on our predictions to generate pseudo
ground-truths to enable adversarial learning which is explained in the next
section.

###  3.3 Adversarial Learning

Training the model with an adversarial loss applied on the rendered images do
not improve the results due to the shortcomings of renderers [40]. Therefore,
we convert texture learning into 2D image synthesis task. Texture learning in
UV space is previously explored with successful results [11, 40, 8]. However,
our set-up is different as we learn a single view image inference, the texture
projection and pseudo ground-truth generations are online in our training, we
do not learn a GAN trained from scratch for texture generation rather we tune
our 3D reconstruction network, and we propose a multi-view training in our
discriminator. While previous methods use different networks for texture
projection and texture generation, we achieve both with the same architecture.
This enables us to improve the generator further and achieve state-of-the-art
results.

As shown in Fig. 3, during our training, we obtain projected texture maps for
the input view ($v1$) and a different view ($v2$). We obtain those by first
predicting 3D meshes from an input view ($v1$) via our generator. The input
images are projected onto the UV map of the predicted mesh template based on
the camera parameters of each image via an inverse rendering. In this process,
mesh predictions are transformed onto 2D screen by projection with camera
parameters. Then transformed mesh coordinates and UV map coordinates are used
in reverse way and real images are projected onto UV map with the renderer.
Visibility masks are also obtained in this set-up. With this set-up, we obtain
a real partial texture (from $v2$) and a conditioning texture map (from $v1$)
to train our discriminator. In this set-up, it is important for the mesh
predictions to be accurate for the correct inverse rendering. That is way we
leave the GAN training to the third stage.

Discriminator. Finally, to provide adversarial feedback, we train a
conditional discriminator. The discriminator is conditioned on the partial
texture view 1. Partial texture view 2 is a real example and the generated
texture is a fake one. For the fake example, we mask the generated texture
with the visibility mask from real example to prevent distribution mismatch.

In traditional image-to-image translation algorithms conditional input and
target images are concatenated and fed to the discriminators. However, in our
case, the images are partially missing and aligning them in input via
concatenation do not provide useful signals. Instead, we use a projection
based discriminator [35] where we process the conditioned input via
convolutional layers and global pooling until the spatial dimension decreases
to $1\times 1$. Again the reason to decrease the dimension is because the
input is partially missing, therefore we want a full receptive field of the
input image while conditioning on the patches.

We dot product the conditional input that is embedded and the discriminator
outputs. This score is added to the final discriminator score. With the multi-
view conditioning, the discriminator does not only consider if the patch is
realistic but also if the predicted texture is consistent with its input pair.
The GAN training is especially important in our set-up since we do not have
consistent multi-view images. The overall objective for the third stage
includes following min-max optimization:

$\displaystyle\min_{\theta_{g}}\max_{\theta_{d}}~{}\mathcal{L}_{second}+\lambda_{adv}\mathcal{L}_{adv}(\theta_{g},\theta_{d})$
(6)

where $\theta_{d}$ and $\theta_{g}$ refer to parameters of the discriminator
and the generator, respectively.

##  4 Experiments

Datasets. First to generate datasets, we use three category-specific StyleGAN
models, one representing a rigid object class, and two representing
articulated class. For car and horse dataset, official models from StyleGAN2
([19]) repository are used. These models are trained on LSUN Car dataset with
5.7M images and LSUN horse dataset with 2M images [54]. We also use a model
trained on a bird class on NABirds dataset [47] with 48k images. The StyleGAN
generated images are aligned for few view-points and those views are annotated
for one example which takes 1 minutes in total. Please refer to Zhang et. al.
[58] for more details in the dataset generation pipeline.

Architectural Details of Generator. Our generator has an encoder-decoder
architecture as shown in Fig. 4. In the encoder, for predicting deformation
and texture maps, the encoder receives $512\times 512$ image. It has 7
convolutional blocks with each convolution layer with $3\times 3$ filters and
stride of $2$. The number of channels of the convolution layers are
$(64,128,256,256,256,128,128)$. There is a ReLU non-linearity after each
convolution layer. The encoder decreases the spatial size to $4\times 4$.The
encoded features ($128\times 4\times 4$) go through a fully connected layer
and after reshaping, we output features with dimension of $512\times 8\times
4$. Note that, we start with width of $8$ and height of $4$. While generating
texture and mesh predictions, we only predict half of the maps in the height
since they are expected to be mostly symmetric in y axis. Later, we flip the
predictions and concatenate with itself to expand the height.

The texture and mesh predictions go through two shared blocks of convolutional
layers at first. In the decoder, each block has two convolutional layers.
There is an adaptive normalization and leaky ReLU after each convolutional
layer. There is also a skip connection from input to the output at each block.
After each block, there is a bilinear interpolation layer to upsample the
feature maps at each layer. The first two blocks has channels of $(512,256)$
which bring the feature maps to a spatial dimension of $32\times 16$. After
that mesh prediction branches out and there is another block with channel size
of $64$ for the mesh prediction branch. After that there is a single
convolutional layer with channel of $3$. The output represents the
deformations (x,y,z) coordinates.

Figure 4: Encoder-decoder based generator architecturet that takes the input
image and sampled $z$ to output texture and 2D deformation map predictions.
Each block represent a convolution layer with channel size. The encoder and
decoder are connected with fully connected layers. Figure 5: Discriminator
architecture. To provide adversarial feedback, we train a projection based
conditional discriminator.

For texture prediction, there are 4 more convolution blocks with channels size
of $(256,256,128,128)$. After these layers, the spatial resolution becomes
$512\times 256$. A reflection symmetry is applied as we flip the texture
predictions in y axis and concatenate it with the original texture
predictions. This results in spatial resolution of $512\times 512$. After that
there is one more convolution block with channel size of $64$ and a final
convolution layer which decreases the number of channels to $3$. They
represent (R,G,B) channels of the texture prediction. The convolutional layers
after symmetry relaxes the symmetry constrains as we do not expect a perfect
symmetry in the texture.

Note that the mesh predictions are also estimated in a convolutional way as a
UV deformation map [40, 2, 8]. Our deformation is a representation on the
function of sphere directly with a fixed surface topology. We sample from the
deformation map for the corresponding vertex locations. We also apply symmetry
on the predicted UV deformation. We use DIB-R [6] as our differentiable
renderer. The renderer takes mesh and texture predictions and output images
for target camera parameters.

We additionally sample a latent vector from normal distribution to provide a
diversity in our predictions. The sampled latent vector is concatenated with
the encoded features via a linear layer. The output is fed to the adaptive
normalization layers in convolutional blocks.

Figure 6: Given input images (1st column), we predict 3D shape, texture, and
render them into the same viewpoint and novel view points for our model. We
also show renderings of state-of-the-art models that are trained on real and
synthetic images. Since, models use different camera parameters, we did not
align the results. However, the results shown from similar viewpoints capture
the behaviour of each model. Unicorn outputs similar 3D shape for different
inputs. Pixelnerf achieves a high quality same view results but the results
are poor from novel views. Dib-r also suffers from the same issue. Ganverse
does not output realistic details. Our model achieves significantly better
results than the previous works while being trained on a synthetic (StyleGAN
generated) dataset.

Architectural Details of Discriminator. As for the architecture of the
discriminator, we use a projection based discriminator [35]. The discriminator
takes the generated and real $3\times 512\times 512$ texture maps, and the
pseudo ground-truth visibility mask as shown in Fig. 5. Generated textures are
also multiplied with the masks to prevent a mismatch between the real-fake
data distributions. We concatenate the input with learnable positional
embeddings on both scales which is omitted from the figure [8]. The
discriminator adopts a multi-scale architecture with two scales one operates
on $32\times 32$ patches, the other $16\times 16$. We also have a conditioning
pathway shown as an embedding network. The embedding network process the
conditioned input via convolutional layers and global pooling until the
spatial dimension decreases to $1\times 1$. We dot product the conditional
input that is embedded to $1\times 1$ resolution and the discriminator
outputs. This score is added to the final discriminator score. This happens
for both scales of discriminators.

Training parameters We train our model on 8 GPUs with batch size of 4 per GPU,
for 100 epochs in total with learning rate of $1^{-4}$. In our loss functions,
we use $\lambda_{lap}=0.5$, $\lambda_{p-sv}=1$, $\lambda_{p-nv}=1$,
$\lambda_{adv}=1$. The discriminators learning rate is set to $10^{-4}$. We
use Adam optimizer for updating both the 3d inference model and discriminator.

TABLE I:  We report results for same view (input and target have the same view) and novel view (input and target have different view). We provide FID scores, LPIPS, MSE, SSIM, and 2D mIOU accuracy predictions and GT. We compare with GanVerse since they are trained on the same dataset. We also provide results of each stage showcasing improvements of progressive training. |  | Same View | Novel View
---|---|---|---
| Method | FID$\Downarrow$ | LPIPS$\Downarrow$ | MSE$\Downarrow$ | SSIM$\Uparrow$ | IoU$\Uparrow$ | FID$\Downarrow$ | LPIPS $\Downarrow$ | MSE$\Downarrow$ | SSIM$\Uparrow$ | IoU$\Uparrow$
Car | GanVerse [58] | 28.04 | 0.1238 | 0.0060 | 0.8683 | 0.92 | 29.59 | 0.1333 | 0.0075 | 0.8599 | 0.93
Ours - Stage I | 10.75 | 0.1011 | 0.0060 | 0.8695 | 0.94 | 11.98 | 0.1091 | 0.0074 | 0.8582 | 0.92
Ours -Stage II | 6.24 | 0.0737 | 0.0039 | 0.9027 | 0.94 | 9.05 | 0.1012 | 0.0072 | 0.8651 | 0.93
Ours - Stage III | 4.56 | 0.0696 | 0.0040 | 0.9039 | 0.94 | 6.92 | 0.0965 | 0.0076 | 0.8632 | 0.93
Bird | GanVerse [58] | 69.32 | 0.0742 | 0.0034 | 0.9230 | 0.82 | 63.89 | 0.0782 | 0.0037 | 0.9202 | 0.80
Ours - Stage I | 69.58 | 0.0763 | 0.0035 | 0.9222 | 0.82 | 63.76 | 0.0764 | 0.0036 | 0.9222 | 0.81
Ours -Stage II | 64.67 | 0.0720 | 0.0030 | 0.9258 | 0.82 | 60.97 | 0.0749 | 0.0036 | 0.9226 | 0.81
Ours - Stage III | 61.21 | 0.0689 | 0.0030 | 0.9231 | 0.83 | 59.08 | 0.0712 | 0.0035 | 0.9201 | 0.82
Horse | GanVerse [58] | 62.38 | 0.1272 | 0.0060 | 0.8852 | 0.78 | 83.82 | 0.1531 | 0.0100 | 0.8642 | 0.77
Ours - Stage I | 83.66 | 0.1395 | 0.0088 | 0.8727 | 0.78 | 76.64 | 0.1442 | 0.0100 | 0.8669 | 0.78
Ours - Stage II | 57.07 | 0.1037 | 0.0059 | 0.8971 | 0.79 | 67.77 | 0.1368 | 0.0101 | 0.8676 | 0.78
Ours - Stage III | 56.83 | 0.1024 | 0.0055 | 0.9017 | 0.79 | 67.30 | 0.1367 | 0.0099 | 0.8684 | 0.78
Figure 7: Qualitative results of our final model on Bird and Horse class.
Given input images (1st row), we predict 3D shape, texture, and render them
into the same view point and novel view points.

Evaluation. We report various metrics on validation datasets. Since we have
multi-view data, we report the scores for both same view and novel view. Same
view results are obtained by rendering the predictions from the same view
given as the input whereas for novel view, we render the predictions from a
different camera view than the input. Same view looks at the fidelity to the
given input whereas novel view measures if the model estimates the invisible
texture and geometry which is a more difficult task. For both views, we report
Frechet Inception Distance (FID) metric [16] which looks at the realism by
comparing the target distribution and rendered images, Learned Perceptual
Image Patch Similarity (LPIPS) [57] which compares the target and rendered
output pairs at the feature level of a pretrained deep network, Structural
Similarity Index Measure (SSIM) and Mean Squared Error (MSE) which compare the
pairs in pixel-level similarity. We also report the intersection-over-union
(IoU) between the target silhouette and projected silhouette of the predicted
geometry.

Results. We provide quantitative results in Table I. We provide comparisons
with GanVerse [58] which is trained on the same data as ours. We observe
quantitative improvements at each stage for all three classes as given in
Table I as well as large improvements over GanVerse model especially on FID
metrics which measure the quality of generations. We qualitatively compare
with other methods, since they all use different camera set-ups, it is not
possible to do an accurate quantitative comparison. However, on our
qualitative result comparisons (Fig. 6), it is clear that our results achieve
significantly better results.

In our qualitative comparisons, we compare with Unicorn [36] which learns 3D
inference model in an unsupervised way on Pascal3D+ Car dataset [51]. The
model only uses the bounding box annotation and is trained on 5000 training
images. As can be seen from Fig. 6, impressive results are achieved given that
the model is learned in an unsupervised way. On the other hand, the results
lack details and diversity in the shapes. It is significantly worse than ours.
Second, we compare with Pixelnerf [53] which predicts a continuous neural
scene representation conditioned on a single view image. While neural radiance
fields [34] optimizes the representation to every scene independently,
Pixelnerf trains across multiple scenes to learn a scene prior and is able to
perform novel view synthesis given an input image. Pixelnerf is trained on a
synthethic dataset, ShapeNet dataset [5]. It is also showcased on real image
reconstruction for car classes. In our results, Pixelnerf is very good at
predicting the same view but not as successful on the novel view predictions.
Next, we compare with Dib-r model [6] which is trained on Pascal3D+ Car
dataset with ground-truth silhouette and camera parameters. Dib-r outputs
reasonable results on the same-view predictions of the input image. However,
their texture and meshes do not generalize across views and results in
unrealistic predictions from novel views even though the model is trained with
expensive annotations.

Figure 8: 3D model predictions of our model.

Last, we compare with GanVerse model which is trained on the same StyleGAN
generated dataset as our method. As shown in Table I, the quantitative results
of GanVerse are even worse than our single-stage results. GanVerse model is
trained with same view and other view reconstructions simultaneously. One
difference is that, GanVerse model learns a mean shape and additional
deformation vertices for each image. The additional deformation is penalized
for each image for stable training. Since, we rely on other view training for
mesh prediction, we do not put such constrain on the vertices. Their
architecture is based on a U-Net and they do not employ a GAN training. Their
results lack details and do not look as realistic compared to our model. Note
that StyleGAN generated dataset also removes the expensive labeling effort and
models that train on this dataset have the same motivation with the models
that learn without annotations such as Unicorn. Generating StyleGAN dataset
with annotations requires 1 minute, whereas Pascal3D+ dataset requires 200h
350h work time for the annotations [58]. Therefore, our comparisons provided
in Fig. 6 cover models learned from a broad range of dataset set-ups with
different levels of annotation efforts. It includes unsupervised training data
(Unicorn with Pascal3D+ images), StyleGAN generated data which adds a minute
longer annotation effort (our method), a much more expensive data with real
images and with key-point annotations (Dib-r on labeled Pascal3D+ images), and
a synthetic data (Pixelnerf with ShapeNet dataset) with perfect annotations.

Lastly, we show the final results of our method on bird and horse classes in
Fig. 7. 3D models of these categories without textures are also shown in Fig.
8. Our method achieves realistic 3D predictions for these classes as well even
though StyleGAN generated datasets have inconsistencies across views.

Figure 9: Validation FID curve on Car class with respect to number of iterations. As shown, the improvements are not coming from longer iterations but from the progressive training. TABLE II: Ablation study showing results without multi-stage pipeline, without multi-view discriminator, and without the gaussian sampling in the generator on Car dataset. Results are given for same and novel views and contains scores, LPIPS, MSE, SSIM, and 2D mIOU accuracy predictions and GT. | Same View | Novel View
---|---|---
Method | FID $\Downarrow$ | LPIPS $\Downarrow$ | MSE $\Downarrow$ | SSIM $\Uparrow$ | IoU $\Uparrow$ | FID $\Downarrow$ | LPIPS $\Downarrow$ | MSE $\Downarrow$ | SSIM$\Uparrow$ | IoU $\Uparrow$
Ours - Stage I | 10.75 | 0.1011 | 0.0060 | 0.8695 | 0.94 | 11.98 | 0.1091 | 0.0074 | 0.8582 | 0.92
Ours -Stage II | 6.24 | 0.0737 | 0.0039 | 0.9027 | 0.94 | 9.05 | 0.1012 | 0.0072 | 0.8651 | 0.93
Ours - Stage III | 4.56 | 0.0696 | 0.0040 | 0.9039 | 0.94 | 6.92 | 0.0965 | 0.0076 | 0.8632 | 0.93
No Multi-Stage | 66.59 | 0.1424 | 0.0118 | 0.8276 | 0.91 | 66.71 | 0.1485 | 0.0136 | 0.8200 | 0.90
Same-View Training | 4.88 | 0.0759 | 0.0037 | 0.9034 | 0.96 | 42.24 | 0.1256 | 0.0105 | 0.8469 | 0.88
Multi-View Training | 6.60 | 0.0741 | 0.0039 | 0.9026 | 0.94 | 10.00 | 0.1017 | 0.0071 | 0.8657 | 0.94
No Multi-View disc. | 4.65 | 0.0702 | 0.0039 | 0.9042 | 0.94 | 7.02 | 0.0965 | 0.0076 | 0.8633 | 0.93
No Gaussian samp. | 4.67 | 0.0696 | 0.0039 | 0.9043 | 0.94 | 7.05 | 0.0986 | 0.0075 | 0.8625 | 0.93

Ablation Study. First, we analyze the role of each stage in our training
pipeline. As given in Table I, additional training at each stage improves
metrics consistently, especially FIDs and LPIPS, the metrics that are shown to
closely correlate with human perception. In Fig. 9, we provide the training
curves of each stage. As can be seen from figure, the improvements are not
coming from longer trainings but from the progressive learning. We provide
qualitative comparisons of each stage’s output renderings in Fig. 10. First
stage rendering outputs results with a reliable geometry. However, the texture
is not realistic, especially tires have duplicated features which is
understandable given that the model is minimizing the reconstruction loss from
inconsistent multi-view images. In the second-stage, texture improves
significantly over first-stage. Finally, with GAN training in the last stage,
the colors look more realistic, sharp, and fine-details are generated.

We provide additional ablation study in Table II conducted on Car dataset. We
also provide our each stage scores in the first block to compare the results
easily. In the second block, we first experiment with no multi-stage training
(No Multi-Stage). This set-up refers to training the model from scratch with
the final proposed loss function. This training results in poor results in all
metrics and even worse than our first-stage training results. Especially,
adversarial training makes the training less stable when the model did not yet
learn reliable predictions. It shows the importance of our multi-stage
training pipeline.

Figure 10: Qualitative results of our models from first, second, and third
stage trainings on Car class. At each stage, results improve significantly.
For example, zooming into tires in the first stage, we see duplicated
features. The second stage solves that problem mostly but results are not as
sharp as the third stage results.

Next, we train with Same-view training objectives. This set-up is used when
multi-view images are not available and models have to be trained on single-
view images. We use a similar perceptual objective to Eq. 4 but with the
ground-truth silhouettes as given in Eq. 7. We also add the silhouette loss
from the same view to guide the geometry predictions. Here the input and
target images share the same camera parameters.

$\small\mathcal{L}_{p-sv-
sil}=||\Phi_{j}(I^{g}_{v1}*S^{g}_{v1})-\Phi_{j}(I^{r}_{v1}*S^{g}_{v1})||$ (7)

$\vspace{-1pt}\small\mathcal{L}_{sil-sv}=1-\frac{||S^{g}_{v1}\odot
S^{r}_{v1}||_{1}}{||S^{g}_{v1}+S^{r}_{v1}-S^{g}_{v1}\odot S^{r}_{v1}||}$ (8)

$\begin{multlined}\small\mathcal{L}_{sv}=\lambda_{pn}\mathcal{L}_{p-sv-
sil}+\lambda_{s}\mathcal{L}_{sil-
sv}+\lambda_{lap}\mathcal{L}_{lap}\end{multlined}\small\mathcal{L}_{sv}=\lambda_{pn}\mathcal{L}_{p-sv-
sil}+\lambda_{s}\mathcal{L}_{sil-sv}+\lambda_{lap}\mathcal{L}_{lap}$ (9)

The same-view training (model trained with objective from Eq. 9) results are
given in Table II. With this set-up, same view results look good
quantitatively because the network learns to reconstruct the input view.
Especially IoU of the same view is better than the other set-ups because the
model learns the missing parts of input images and their corresponding
silhouettes and make similar predictions on the validation dataset that match
the ground-truth. On the other hand, IoU of novel view score is the worst
among all set-ups. Models trained with single-view objectives struggle
generating realistic novel views as can be seen in FIDs, 42.24 novel view FID
versus 4.88 same view FID.

We also experiment with multi-view training set-up. This refers to training
the model from scratch with the second stage objective. We compare those
results with our second-stage training results. We see that better results are
achieved with the progressive learning.

In the last block, we run experiments where we train a discriminator without
the multi-view conditioning. For this experiment, we only update the stage
three training. The discriminator only has the main pipeline without the
projection based other view conditioning. This also results in worse results
than our proposed multi-view conditional discriminator, especially in FIDs.
Since the multi-view conditional discriminator receives guidance from a given
view, it propagates better signals to the generator.

We also experiment the setting which does not have the sampling from normal
distribution. This change converts the setting to a deterministic model. We
observe that sampling provides with a slight diversity in the colors and
improves the metrics slightly so we decide to keep it. The diversity results
are given in Fig. 11. The model does not achieve visible diversity, however,
when we take the difference of two images, they are slightly different. The
improvements are not significant but consistent across all metrics.

|  |
---|---|---
Sample 1 | Sample 2 | Difference Map
|  |
Sample 1 | Sample 3 | Difference Map
|  |
Sample 2 | Sample 3 | Difference Map

Figure 11: Texture predictions with different sampled codes and with same
input image. The model does not achieve visible diversity. However, when we
take the difference of two images, they are slightly different.

##  5 Conclusion

In this work, we present a method to reconstruct high-quality textured 3D
models from single images. Our method learns from GAN generated images and
bypasses the reliance on labeled multi-view datasets or expensive 3D scans.
GAN generated dataset is labeled in mass and requires a total of 1 minute
human-labor. Because GAN generated dataset is noisy and not strictly
consistent across views, we propose a novel multi-stage training pipeline and
adversarial training set-up. We achieve significant improvements over previous
methods whether they were trained on GAN generated images or on real images.

Limitations. First limitation of our work is that we deform our final meshes
from a sphere and cannot handle objects with holes similar to previous works
that build their work on mesh representations obtained by deforming from
spheres [17, 58, 40]. Another limitation we observe is the different 3D model
qualities we obtain across different categories. Specifically, our generated
3D models are better quality for the car class than the bird class. We also
observe the same on the StyleGAN image generations results between car and
bird classes. One reason for that is that StyleGAN model is trained on 5.7M
car images whereas for bird category it is only trained on 48k bird images. We
acknowledge that the performance of our model is correlated with the
performance of the StyleGAN model it learns from. Even though, our model does
not need annotated images, StyleGAN model requires a large amount of unlabeled
data. Learning GAN models on limited data is an important future direction for
this work [18].

## References

  * [1] Y. Alaluf, O. Tov, R. Mokady, R. Gal, and A. Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18511–18521, 2022.
  * [2] A. Bhattad, A. Dundar, G. Liu, A. Tao, and B. Catanzaro.  View generalization for single image textured 3d models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6081–6090, June 2021.
  * [3] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  * [4] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5799–5809, 2021.
  * [5] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [6] W. Chen, H. Ling, J. Gao, E. Smith, J. Lehtinen, A. Jacobson, and S. Fidler.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  In Advances in Neural Information Processing Systems, pages 9609–9619, 2019.
  * [7] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In European conference on computer vision, pages 628–644. Springer, 2016.
  * [8] A. Dundar, J. Gao, A. Tao, and B. Catanzaro.  Fine detailed texture learning for 3d meshes with generative models.  arXiv preprint arXiv:2203.09362, 2022.
  * [9] A. Dundar, K. Sapra, G. Liu, A. Tao, and B. Catanzaro.  Panoptic-based image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8070–8079, 2020.
  * [10] J. Gao, T. Shen, Z. Wang, W. Chen, K. Yin, D. Li, O. Litany, Z. Gojcic, and S. Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  arXiv preprint arXiv:2209.11163, 2022.
  * [11] B. Gecer, S. Ploumpis, I. Kotsia, and S. Zafeiriou.  Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1155–1164, 2019.
  * [12] S. Goel, A. Kanazawa, and J. Malik.  Shape and viewpoint without keypoints.  arXiv preprint arXiv:2007.10982, 2020.
  * [13] J. Gu, L. Liu, P. Wang, and C. Theobalt.  Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis.  arXiv preprint arXiv:2110.08985, 2021.
  * [14] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [15] P. Henderson, V. Tsiminaki, and C. H. Lampert.  Leveraging 2d data to learn textured 3d mesh generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7498–7507, 2020.
  * [16] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [17] A. Kanazawa, S. Tulsiani, A. A. Efros, and J. Malik.  Learning category-specific mesh reconstruction from image collections.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 371–386, 2018.
  * [18] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila.  Training generative adversarial networks with limited data.  Advances in neural information processing systems, 33:12104–12114, 2020.
  * [19] T. Karras, S. Laine, and T. Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [20] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [21] H. Kato, Y. Ushiku, and T. Harada.  Neural 3d mesh renderer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3907–3916, 2018.
  * [22] J. Ko, K. Cho, D. Choi, K. Ryoo, and S. Kim.  3d gan inversion with pose optimization.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2967–2976, 2023.
  * [23] A. Lattas, S. Moschoglou, B. Gecer, S. Ploumpis, V. Triantafyllou, A. Ghosh, and S. Zafeiriou.  Avatarme: Realistically renderable 3d facial reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 760–769, 2020.
  * [24] X. Li, S. Liu, K. Kim, S. De Mello, V. Jampani, M.-H. Yang, and J. Kautz.  Self-supervised single-view 3d reconstruction via semantic consistency.  arXiv preprint arXiv:2003.06473, 2020.
  * [25] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin.  Magic3d: High-resolution text-to-3d content creation.  arXiv preprint arXiv:2211.10440, 2022.
  * [26] F. Liu and X. Liu.  2d gans meet unsupervised single-view 3d reconstruction.  arXiv preprint arXiv:2207.10183, 2022.
  * [27] G. Liu, A. Dundar, K. J. Shih, T.-C. Wang, F. A. Reda, K. Sapra, Z. Yu, X. Yang, A. Tao, and B. Catanzaro.  Partial convolution for padding, inpainting, and image synthesis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [28] S. Liu, T. Li, W. Chen, and H. Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7708–7717, 2019.
  * [29] Y. Liu, Z. Shu, Y. Li, Z. Lin, R. Zhang, and S. Kung.  3d-fm gan: Towards 3d-controllable face manipulation.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pages 107–125. Springer, 2022.
  * [30] M. M. Loper and M. J. Black.  Opendr: An approximate differentiable renderer.  In European Conference on Computer Vision, pages 154–169. Springer, 2014.
  * [31] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210–7219, 2021.
  * [32] G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  arXiv preprint arXiv:2211.07600, 2022.
  * [33] O. Michel, R. Bar-On, R. Liu, S. Benaim, and R. Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13492–13502, 2022.
  * [34] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [35] T. Miyato and M. Koyama.  cgans with projection discriminator.  In International Conference on Learning Representations, 2018.
  * [36] T. Monnier, M. Fisher, A. A. Efros, and M. Aubry.  Share with thy neighbors: Single-view reconstruction by cross-instance consistency.  arXiv preprint arXiv:2204.10310, 2022.
  * [37] T. Nguyen-Phuoc, C. Li, L. Theis, C. Richardt, and Y.-L. Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7588–7597, 2019.
  * [38] M. Niemeyer and A. Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [39] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.
  * [40] D. Pavllo, G. Spinks, T. Hofmann, M.-F. Moens, and A. Lucchi.  Convolutional generation of textured 3d meshes.  arXiv preprint arXiv:2006.07660, 2020.
  * [41] H. Pehlivan, Y. Dalva, and A. Dundar.  Styleres: Transforming the residuals for real image editing with stylegan.  arXiv preprint arXiv:2212.14359, 2022.
  * [42] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [43] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv preprint arXiv:2007.08501, 2020.
  * [44] N. Tritrong, P. Rewatbowornwong, and S. Suwajanakorn.  Repurposing gans for one-shot semantic part segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4475–4485, 2021.
  * [45] S. Tulsiani, A. A. Efros, and J. Malik.  Multi-view consistency as supervisory signal for learning shape and pose prediction.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2897–2905, 2018.
  * [46] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik.  Multi-view supervision for single-view reconstruction via differentiable ray consistency.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2626–2634, 2017.
  * [47] G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595–604, 2015.
  * [48] T. Wang, Y. Zhang, Y. Fan, J. Wang, and Q. Chen.  High-fidelity gan inversion for image attribute editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11379–11388, 2022.
  * [49] T.-C. Wang, A. Mallya, and M.-Y. Liu.  One-shot free-view neural talking-head synthesis for video conferencing.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10039–10049, 2021.
  * [50] Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu.  Nerf–: Neural radiance fields without known camera parameters.  arXiv preprint arXiv:2102.07064, 2021.
  * [51] Y. Xiang, R. Mottaghi, and S. Savarese.  Beyond pascal: A benchmark for 3d object detection in the wild.  In IEEE winter conference on applications of computer vision, pages 75–82. IEEE, 2014.
  * [52] F. Yin, Y. Zhang, X. Wang, T. Wang, X. Li, Y. Gong, Y. Fan, X. Cun, Y. Shan, C. Oztireli, et al.  3d gan inversion with facial symmetry prior.  arXiv preprint arXiv:2211.16927, 2022.
  * [53] A. Yu, V. Ye, M. Tancik, and A. Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [54] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [55] N. Yu, G. Liu, A. Dundar, A. Tao, B. Catanzaro, L. S. Davis, and M. Fritz.  Dual contrastive loss and attention for gans.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6731–6742, 2021.
  * [56] J. Y. Zhang, P. Felsen, A. Kanazawa, and J. Malik.  Predicting 3d human dynamics from video.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7114–7123, 2019.
  * [57] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [58] Y. Zhang, W. Chen, H. Ling, J. Gao, Y. Zhang, A. Torralba, and S. Fidler.  Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering.  International Conference on Learning Representations, 2020.
  * [59] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145–10155, 2021.


# Carrier-induced modulation of radiation by a gated graphene

M.V. Strikha  F.T. Vasko<EMAIL_ADDRESS>Institute of Semiconductor
Physics, NAS of Ukraine, Pr. Nauky 41, Kyiv, 03028, Ukraine

###### Abstract

The modulation of the transmitted (reflected) radiation due to change of
interband transitions under variation of carriers concentration by the gate
voltage is studied theoretically. The calculations were performed for strongly
doped graphene on high-$\kappa$ (Al2O3, HfO2, AlN, and ZrO2) or SiO2
substrates under normal propagation of radiation. We have obtained the
modulation depth above 10% depending on wavelength, gate voltage (i.e.
carriers concentration), and parameters of substrate. The graphene -
dielectric substrate - doped Si (as gate) structures can be used as an
effective electrooptical modulator of near-IR and mid-IR radiation for the
cases of high-$\kappa$ and SiO2 substrates, respectively.

###### pacs:

78.67.Wj, 42.79.Hp

##  I Introduction

The essential feature of graphene’s optical properties is its substantial
interaction with radiation in the wide spectral region, from far-$IR$ up to
$UV$, due to effective interband transitions (see reviews 1 ). The enhancement
of this responce due to interference permits one to make the graphene on
dielectric substrate visible 2 . The other example of the exceptional optical
properties is the graphene-based saturable absorber for ultrafast lasers in
the telecommunication spectral region 3 . Besides this, the carriers
contribution modifies essentially the graphene response due to the Pauli
blocking effect, when absorption is supressed at
$\hbar\omega/2<\varepsilon_{F}$, where $\omega$ is the frequency of radiation,
$\varepsilon_{F}$ is the Fermy energy, see experimental data and discussion in
Refs. 4 and 5. Recently, modulation 5a  and polarization 5b  of IR radiation
were observed in the graphene structure integrated with an optical waveguide.
The efficiency of modulation can also be enhanced for the case of normal
propagation of radiation by the interference effect under an appropriate
thickness of substrate, see Figs. 1a and 1b. As a result, an improvement of
modulation for transmission and reflection coefficients of the graphene -
substrate - gate structure by a gate voltage takes place in contrast to the
case of graphene placed on a semi-infinete insulator. 5c  In this paper, we
perform the calculation of the optical characteristics of such a structure,
and we discuss the conditions for realisation of the graphene-based modulator
in the near- and mid-IR spectral region.

Figure 1: (Color online) (a) Schematic view of graphene - high-$\kappa$
substrate - heavily doped Si structure ($V_{g}$ is gate voltage). (b) Normal
propagation of incident ($in$), reflected ($r$), and transmitted ($t$)
radiation through the graphene placed over substrate of thickness $d$ on doped
Si ($\epsilon_{s}$ and $\epsilon$ are the correspondent dielectric
permittivities). (b) Threshold wavelenthes $\lambda_{th}$ versus transverse
field $E_{\bot}$ for structures with different substrates: SiO2 (1), AlN (2),
Al2O3 (3), and HfO2 (4) (the results for ZrO2 are similar to curve (4)).
Dashed line is correspondent to the telecommunication wavelength, 1.55 $\mu$m.
(c) Spectral dependences of Re$\sigma_{\omega}$ and Im$\sigma_{\omega}$ (solid
and dotted curves) at room temperature for concentrations: 1012 cm-2 (1),
$2\times 10^{12}$ cm-2 (2), $4\times 10^{12}$ cm-2 (3).

At low temperatures, or high doping levels, the threshold frequency for the
jump of absorption is determined by the condition
$\hbar\omega_{th}=2\varepsilon_{F}\propto\sqrt{n}$, where concentration $n$
depends linearly on gate voltage, $V_{g}$, and is inverse to substrate
thickness, $d$, so that $n\propto V_{g}/d\equiv E_{\bot}$. The dependences of
the threshold wavelength $\lambda_{th}$ on homogeneous field $E_{\bot}$ are
presented in Fig. 1c for a number of substrates. One can see that the
modulation of near-IR radiation (1.55 $\mu$m) is possible at $E_{\bot}<$3
MV/cm for the high-$\kappa$ substrates (Al2O3, HfO2, AlN, and ZrO2 are
examined below), and for SiO2 substrate the applied field should be twice
stonger. The modulation of mid-IR radiation (10.6 $\mu$m) can be realised in
lower fieids, $E_{\bot}<$ 1 MV/cm. For the modulation of radiation in the
visible spectral range, a field $E_{\bot}$ comparable with the breakdown field
for the substrate under consideration is needed. The modulation depth can be
estimated through the amplitude of the absorbtion jump on the threshold, equal
to 2.3%. For the 5 layer graphene the modulation efficiency can exceed 10%,
taking into account that the interference on substrate influences this value.
The absorption edge spreads with the increase of temperature (see the real
part of dynamic conductivity $\sigma_{\omega}$ presented in Fig. 1d), and the
modulation efficiency decreases. At room temperature the effective modulation
for near-IR radiation only is possible, while the modulation of mid-IR
radiation needs cooling to temperatures about 77 K. The efficiency of
modulation obtained is comparable to results both in bulk semiconductors and
in heterostructures, see Refs. 9 and 10, respectively.

The analysis performed below is organized as follows. In Sect. II we describe
the carrier-induced modulation of the response of graphene and evaluate the
transmission and reflection coefficients. The efficiency of modulation versus
applied field and thickness of substrate is analyzed in Sect. III. Discussion
and concluding remarks are given in the last section.

##  II Basic equations

The response of a doped graphene sheet on probe radiation is described by the
two-dimensional dynamic conductivity $\sigma_{\omega}$. In the colisionless
approximation, when $\omega$ exceeds any relaxation rate, the real and
imaginary parts of $\sigma_{\omega}$ are given by the expressions:

$\displaystyle{\rm
Re}\sigma_{\omega}\simeq\frac{e^{2}}{4\hbar}(1-f_{ep_{\omega}}-f_{hp_{\omega}}),$
(1) $\displaystyle{\rm
Im}\sigma_{\omega}\simeq\bar{\sigma}_{\omega}-\frac{e^{2}}{4\hbar
p_{\omega}}{\cal
P}\int\limits_{0}^{\infty}\frac{dpp^{2}}{p_{\omega}^{2}-p^{2}}\left(f_{ep}+f_{hp}\right).$

In a doped graphene, when $E_{\bot}$ is strong enough, the Pauli blocking
factor in ${\rm Re}\sigma_{\omega}$ is written through electron and hole
distribution functions, $f_{e,h~{}p}$, taken at $p_{\omega}=\hbar\omega/(2v)$,
where $v=10^{8}$ cm/s is the velocity of quasiparticles. Mention, that in the
absence of carriers ${\rm Re}\sigma_{\omega}$ does not depend on any material
parameters. 6  In the imagionary part of $\sigma_{\omega}$ we eliminate the
contribution of the non-doped graphene $\bar{\sigma}_{\omega}$, see Refs. 12
and 13, and the carriers contribution is connected with ${\rm
Re}\sigma_{\omega}$ through the Kramers-Kronig relation, where $\cal P$ means
the principal value of integral. The response of $N$-layer epitaxial graphene
9  is described by the total conductivity $N\sigma_{\omega}$.

We restrict ourselvers to the geometry of normal propagation of the incident
($in$-), reflected ($r$-), and transient ($t$-) waves through the structure
”N-layer graphene - dielectric substrate - doped Si”, as it is shown in Fig.
1a. The in-plane electric field $E(z)\exp(-i\omega t)$ is governed by the wave
equation: 1 ; 8

$\frac{d^{2}E(z)}{dz^{2}}+\epsilon_{\omega}(z)\left(\frac{\omega}{c}\right)^{2}E(z)=0,$
(2)

where $z\neq 0$ and the permittivity $\varepsilon_{\omega}(z)$ is equal to the
constant $\varepsilon_{s}$ in the substrate layer with the thickness $d$ (at
$0<z<d$), while in the thick Si the dispersion $\varepsilon_{\omega}$ should
be taken into account, see Ref. 15. The boundary conditions at $N$-layer
graphene sheet, where $z\rightarrow 0$, takes the form

$\left.\frac{dE(z)}{dz}\right|_{-0}^{+0}+i\frac{4\pi\omega}{c^{2}}N\sigma_{\omega}E(z=0)=0,~{}~{}~{}~{}\left.E(z)\right|_{-0}^{+0}=0.$
(3)

These expressions contain the contribution of surface current, proportional to
$N\sigma_{\omega}$, that determines the jump of $[dE(z)/dz]$, while $E(z)$ is
continuous. At the substrate-Si interface we use the two conditions of
continuity:

$\left.\frac{dE(z)}{dz}\right|_{d-0}^{d+0}=0,~{}~{}~{}~{}E(z)|_{d-0}^{d+0}=0.$
(4)

Outside of the graphene sheet the solution of electrodynamic problem (2)-(4)
should be written in the form

$E(z)=\left\\{\begin{array}[]{*{20}c}E_{in}e^{ik_{\omega}z}+E_{r}e^{-ik_{\omega}z}&z<0\\\
E_{+}e^{i\tilde{k}_{\omega}z}+E_{-}e^{-i\tilde{k}_{\omega}z}&0<z<d\\\
E_{t}e^{i\bar{k}_{\omega}z}&d<z\end{array}\right..$ (5)

Here the amlitudes for $in$\- and $r$-waves ($z<0$), $t$-wave ($z>d$), and for
the field $E_{\pm}$ in the dielectric substrate (at $0<z<d$) are introduced.
In Eq.(5) the wave vectors $k_{\omega}=\omega/c$ (to the left),
$\tilde{k}_{\omega}=\sqrt{\epsilon_{s}}\omega/c$ (to the right), and
$\bar{k}_{\omega}=\sqrt{\epsilon_{\omega}}\omega/c$ (in the dielectric
substrate) are introduced as well. Using the boundary conditions (3) and (4)
we get the system of linear equations for the amlitudes above. The solution of
such a system determines the transition and reflection coefficients,
$T_{\omega}$ and $R_{\omega}$, according to

$T_{\omega}=\sqrt{\epsilon_{\omega}}\frac{|E_{t}|^{2}}{E_{in}^{2}},~{}~{}~{}~{}~{}R_{\omega}=\frac{|E_{r}|^{2}}{E_{in}^{2}}.$
(6)

According to energy conservation law, which connects $T_{\omega}$ and
$R_{\omega}$ with the relative absorption coefficient $\xi_{\omega}$, one
obtains:

$T_{\omega}+R_{\omega}+\xi_{\omega}=1.$ (7)

As a result, variations of $T_{\omega}$ and $R_{\omega}$ are correlated due to
the Pauli blocking effect which leads to a jump of $\xi_{\omega}$.

The direct expressions for the transmission and reflection coefficients take
form

$T_{\omega}=\frac{4\sqrt{\epsilon_{\omega}}}{|{A_{\omega}^{(+)}}|^{2}},~{}~{}~{}~{}~{}~{}R_{\omega}=\left|\frac{A_{\omega}^{(-)}}{A_{\omega}^{(+)}}\right|^{2}.$
(8)

where $A^{(\pm)}_{\omega}$ are expressed through the dynamic conductivity and
the structure parameters according to

$\displaystyle
A_{\omega}^{(\pm)}=\sqrt{\epsilon}\cos\tilde{k}_{\omega}d-i\sqrt{\epsilon_{s}}\sin\tilde{k}_{\omega}d$
(9) $\displaystyle+\left(\frac{4\pi\sigma_{\omega}}{c}\pm
1\right)\left({\cos\tilde{k}_{\omega}d-i\sqrt{\varepsilon/\varepsilon_{s}}\sin\tilde{k}_{\omega}d}\right).$

In the case $\epsilon_{\omega}=\epsilon_{s}$ the oscillating factors eliminate
from (9), and Eqs. (8) are in agreement with the previous results. 1 ; 8
Taking into consideration the interference on the substrate, when
$\epsilon_{\omega}\neq\epsilon_{s}$, the spectral dependences of $T_{\omega}$
and $R_{\omega}$ are determined by carriers concentration (through variations
of $E_{\bot}$ or $V_{g}$), the dielectric substrate thickness $d$, and the
permittivities $\varepsilon_{s}$ and $\varepsilon_{\omega}$. We have neglected
a weak absorption in Si and used $\varepsilon_{s}$ for SiO2 and high-$\kappa$
dielectrics from Refs. 15 and 16, respectively.

##  III Results

Performing the numerical integration in Eq. (1) and using Eqs. (8, 9) one
obtains the transmission and reflection coefficients. Below we analyze the
dependences of $T_{\omega}$ and $R_{\omega}$ on the applied field $E_{\bot}$,
which determines the carriers concentration, for the substrates of various
thickness on the base of high-$\kappa$ dielectrics or SiO2. The computations
were performed for near-IR and mid-IR spectral regions ($\lambda=$1.55 $\mu$m
and 10.6 $\mu$m).

Figure 2: (Color online) (a) Contour plot of transmission coefficient of
graphene over Al2O3 substrate as a function of applied field $E_{\bot}$ and
thickness $d$. (b) The same for reflection coefficient.

###  III.1 High-$\kappa$ substrates

We examine first the modulation of the telecommunication range radiation,
$\lambda=$1.55 $\mu$m, by the structures of 5-layer graphene on high-$\kappa$
substrate at room temperature. Figures 2(a) and 2(b) show the contour plots of
$T$ and $R$ versus $E_{\bot}$ and $d$ for the case of Al2O3 substrate. One can
see, that the change of $T$ versus $E_{\bot}$ is $\sim$10% near the
transmission maximum, and the modification of $T$ by interference can be as
large as 0.3 if $d$ is in the range 0.1 - 0.5 $\mu$m. Similarly, the change of
$R$ versus $E_{\bot}$ does not exceed several %, while the modification of $R$
versus $d$ can be of 0.3 order. The maximum of $T$ corresponds the minimum of
$R$ and vise versa.

Figure 3: (Color online) Transmissivity $T$ at wavelength 1.55 $\mu$m versus
$E_{\bot}$ for different substrates: (a) Al2O3, (b) HfO2, (c) AlN, and (d)
ZrO2. Curves 1 - 5 are correspondent to the thicknesses $d=$0.08, 0.12, 0.16,
0.2, and 0.24 $\mu$m.  Figure 4: (Color online) Reflectivity $R$ at wavelength
1.55 $\mu$m versus $E_{\bot}$ for the same substrates as in Fig. 3(a-d).
Curves 1 - 5 are correspondent to the thicknesses $d=$0.28, 0.32, 0.36, 0.4
and 0.44 $\mu$m.

Gate-voltage-induced modification of transmission is presented in Fig. 3 for
different high-$\kappa$ substrates at several thicknesses near the maximum of
$T$. Similar dependences for $R$ are presented in Fig. 4 near the reflection
maximum, corresponding to the greater thicknesses. Besides the essential
dependence on thickness, $T$ and $R$ depend also on high-frequency and static
permittivity of the materials under consideration. Therefore, the effective
modulation for Al2O3 and AlN occurs in the range $E_{\bot}\sim$8 - 12 MV/cm,
and for HfO2 and ZrO2 the weaker fields $E_{\bot}\sim$4 - 6 MV/cm are needed.
The modulation depth for transmission exceeds in several times the modulation
depth for reflection. The effective modulation interval, corresponding the
region of the jump in absorption, becomes narrower with the decrease of
temperature.

###  III.2 SiO2 substrate

Now we are going to examine the structures ”graphene - SiO2 \- Si”, where the
permittivities are smaller. Therefore the effective modulation for
transmission of near-IR radiation takes place at $E_{\bot}\sim$25 - 35 MV/cm,
see Fig. 5(a), i.e. it needs stronger fields than the threshold of single-
layer graphene on SiO2 substrate $\sim$6 MV/cm, see Fig. 1(b). The modulation
of $R$ in the same range of fields does not exceed several percents, see Fig.
5(b). It should be noted, that these fields $E_{\bot}$ are of the same order
of values, as a breakdown field, therefore a possibility of modulation in this
case needs a special verification.

Figure 5: (Color online) Transmissivity (a) and reflectivity (b), $T$ and $R$,
at wavelength 1.55 $\mu$m versus $E_{\bot}$ for SiO2 substrate. Curves 1 - 5
are correspondent to the thicknesses $d=$0.24, 0.28, 0.32, 0.36 and 0.4
$\mu$m.

The effective modulation of transmission (over 5%, see Fig. 6(a)) takes place
in mid-IR spectral region, for $\lambda=$10.6 $\mu$m. The applied field in
this case does not exceed 2 MV/cm, but the substrate thickness should be
greater because of the increase of $\lambda$. The corresponding modulation of
reflection does not exceed several percents as well, see Fig. 6(b).

Figure 6: (Color online) Transmissivity (a) and reflectivity (b) versus
$E_{\bot}$ for five-layer graphene over SiO2 substrate at wavelength 10.6
$\mu$m and room temperature. Curves 1 - 8 are correspondent to the thicknesses
$d=$0.3, 0.6, 0.9, 1.2, 1.5, 1.8, 2.1 and 2.4 $\mu$m.

For the single layer graphene the modulation depth obviously can not be
greater, than 2.3%. However, this modulation of transmission occurs for much
lower fields, than in the previous cases under examination, 200 kV/cm, and the
jump region becomes rather narrow with the decrease of temperature, see Figs.
7a, b.

Figure 7: (Color online) Transmissivity at wavelength 10.6 $\mu$m versus
$E_{\bot}$ for a sigle layer graphene over SiO2 substrate at temperatures 77 K
(a) and 20 K (b). Curves 1 - 8 are correspondent to the same thicknesses as in
Fig. 6.

##  IV Concluding remarks

The results obtained clearly demonstrate the possibility for realization of
the modulator for telecommunication spectral range ($\sim$1.5 $\mu$m) on the
base of multilayer graphene, placed over high-$\kappa$ substrate. The
effective modulation can be realized in this case for the applied fields
$\sim$5 MV/cm, while for the case of SiO2 substrate the field should be
$\sim$20 MV/cm comparable to a breakdown value. The modulation depth for
multi-layer ($N=$5-10) graphene can be as large as 10-20% ($\sim$2% per one
layer). The same efficiency of modulation for the mid-IR radiation ($\sim$10.6
$\mu$m) can be realized for the applied fields not stronger than 2 MV/cm for
the low temperature region.

The consideration performed takes into account the contribution of interband
transitions, described by the complex dynamic conductivity, and the radiation
interference on the structure ”vacuum - graphene - substrate - doped Si” for
the case of normal propagation of radiation. The modulation is determined by
the Pauli blocking effect under the change of the carriers concentration by
the gate voltage, therefore the time of modulation is governed by the
recombination time of the excess concentration of carriers, or by the time of
injection from contacts.

Next, we discuss the assumptions used in our calculations, which are rather
standart ones. The dynamic conductivity of the carriers in the spectral region
under examination is described properly with the use of the linear dispersion
law of carriers in graphene. The phenomenological description of the
dispersion of $Im\sigma_{\omega}$ due to the transitions from the valence band
(see Refs. 12 and 13) does not change the results essentially due to the
smallness of its contribution for the spectral range under consideration. The
study of the declined propagation of radiation is more complicated, and the
modulation efficiency in this case decreases. Moreover, the modulation
efficiency can also be reduced in mid-IR range due to the absorption of
radiation in the doped Si. It should be noted, that modulation of electron
concentration in the gate gives a weak contribution to IR response and can be
neglected in comparison to the Pauli blocking effect in graphene.

In conclusion, the results obtained should stimulate the experimental study of
the electrooptical modulation of the near-IR radiation by the structure of
multilayer graphene over high-$\kappa$ dielectric substrate at room
temperature and high gate voltages (concentrations). For the mid-IR spectral
region the effective modulation can be realized at low temperatures.

###### Acknowledgements.

This work was supported by the State Fundamental Research Fund of Ukraine
(Grant 40.2/069).

## References

  * (1) M. Orlita and M. Potemski, Semicond. Sci. Technol. 25 063001 (2010); F. Bonaccorso, Z. Sun, T. Hasan and A. C. Ferrari, Nature Photonics 4, 611 (2010); L.A. Falkovsky, Phys. Usp. 51, 887 (2008).
  * (2) P. Blake, K. S. Novoselov, A. H. Castro Neto, D. Jiang, R. Yang, T. J. Booth, A. K. Geim, and E. W. Hill, Appl. Phys. Lett. 91, 063124 (2007); D. S. L. Abergel, A. Russell, and V. I. Falko, Appl. Phys. Lett. 91, 063125 (2007).
  * (3) Z. Sun, D. Popa, T. Hasan, F. Torrisi, F. Wang, E. J. R. Kelleher, J. C. Travers, V. Nicolosi, and A. C. Ferrari, Nano Res. 3, 653 (2010); F.T. Vasko, Phys. Rev. B 82, 245422 (2010).
  * (4) Z. Q. Li, E. A. Henriksen, Z. Jiang, Z. Hao, M. C. Martin, P. Kim, H. L. Stormer, and D. N. Basov, Nature Physics 4, 532 (2008).
  * (5) F. Wang, Y. Zhang, C. Tian, C. Girit, A. Zettl, M. Crommie, and Y. R. Shen, Science 320, 206 (2008).
  * (6) M. Liu, X. Yin, E. Ulin-Avila, B. Geng, T. Zentgraf, L. Ju, F. Wang, and X. Zhang, Nature, 474, 64 (2011).
  * (7) Q. Bao, H. Zhang, B. Wang, Z. Ni, C. H. Y. X. Lim, Y. Wang, D. Y. Tang, and K. P. Loh, Nature Photonics 5, 411 (2011).
  * (8) L. A. Falkovsky and S. S. Pershoguba, Phys. Rev. B 76, 153410 (2007); L. A. Falkovsky, Phys. Usp. 51, 887 (2008).
  * (9) M. Cardona, Modulation Spectroscopy (Academic Press, New York, 1969).
  * (10) F.T. Vasko and A.V. Kuznetsov, Electron States and Optical Transitions in Semiconductor Heterostructures (Springer, New York, 1998).
  * (11) T. Stauber, N. M. R. Peres, and A. K. Geim, Phys. Rev. B78, 085432 (2008).
  * (12) M. Bruna and S. Borini, Appl. Phys. Lett. 94 031901 (2009).
  * (13) M. V. Strikha and F.T. Vasko, Phys. Rev. B 81, 115413 (2010).
  * (14) W. A. de Heer, C. Berger, X. Wu, M. Sprinkle, Y. Hu, M. Ruan, J. A. Stroscio, P. N. First, R. Haddon, B. Piot, C. Faugeras, M. Potemski, J.-S. Moon, J. Phys. D: Appl. Phys. 43 374007 (2010).
  * (15) C. M. Herzinger, B. Johs, W. A. McGahan, J. A. Woollam, and W. Paulson J. Appl. Phys. 83, 3323 (1998).
  * (16) M. V. Fischetti, D. A. Neumayer, E. A. Cartier, J. Appl. Phys. 90, 4587 (2001).