first-style=long-short, list/sort=true, make-links=false,
list/template=longtable, list/heading=chapter* PV short = PV, long =
PhotoVoltaic, tag = acronyms CASH short = CASH, long = Combined Algorithm
Selection and Hyperparameter optimization, tag = acronyms MSE short = MSE,
long = Mean Squared Error, tag = acronyms MAE short = MAE, long = Mean
Absolute Error, tag = acronyms nMAE short = nMAE, long = normalized Mean
Absolute Error, tag = acronyms MLP short = MLP, long = Multi-Layer Perceptron,
tag = acronyms GBM short = GBM, long = Gradient Boosting Machine, tag =
acronyms RF short = RF, long = Random Forest, tag = acronyms SVR short = SVR,
long = Support Vector Regression, tag = acronyms ReLU short = ReLU, long =
Rectified Linear Unit, tag = acronyms TanH short = TanH, long = Tangent
Hyperbolic, tag = acronyms

# AutoPV: Automated photovoltaic forecasts with limited information using an
ensemble of pre-trained models

Stefan Meisenbacher
Institute for Automation and Applied Informatics
Karlsruhe Institute of Technology
Eggenstein-Leopoldshafen, 76344, Germany
<EMAIL_ADDRESS>
& Benedikt Heidrich
Institute for Automation and Applied Informatics
Karlsruhe Institute of Technology
Eggenstein-Leopoldshafen, 76344, Germany
&Tim Martin
Institute for Automation and Applied Informatics
Karlsruhe Institute of Technology
Eggenstein-Leopoldshafen, 76344, Germany
& Ralf Mikut
Institute for Automation and Applied Informatics
Karlsruhe Institute of Technology
Eggenstein-Leopoldshafen, 76344, Germany
& Veit Hagenmeyer
Institute for Automation and Applied Informatics
Karlsruhe Institute of Technology
Eggenstein-Leopoldshafen, 76344, Germany

###### Abstract

Accurate PV power generation forecasting is vital for the efficient operation
of Smart Grids. The automated design of such accurate forecasting models for
individual PV plants includes two challenges: First, information about the PV
mounting configuration (i.e. inclination and azimuth angles) is often missing.
Second, for new PV plants, the amount of historical data available to train a
forecasting model is limited (cold-start problem). We address these two
challenges by proposing a new method for day-ahead PV power generation
forecasts. The proposed AutoPV method is a weighted ensemble of forecasting
models that represent different PV mounting configurations. This
representation is achieved by pre-training each forecasting model on a
separate PV plant and by scaling the model’s output with the peak power rating
of the corresponding PV plant. To tackle the cold-start problem, we initially
weight each forecasting model in the ensemble equally. To tackle the problem
of missing information about the PV mounting configuration, we use new data
that become available during operation to adapt the ensemble weights to
minimize the forecasting error. The proposed AutoPV method is advantageous as
the unknown PV mounting configuration is implicitly reflected in the ensemble
weights, and only the PV plant’s peak power rating is required to re-scale the
ensemble’s output. The ensemble approach also allows to represent PV plants
with panels distributed on different roofs with varying alignments, as these
mounting configurations can be reflected proportionally in the weighting.
Additionally, the required computing memory is decoupled when scaling AutoPV
to several hundreds of PV plants, which is beneficial in Smart Grid
environments with limited computing capabilities. For a real-world data set
with 11 PV plants, the accuracy of AutoPV is comparable to an individual model
trained on two years of data and outperforms an incrementally trained
individual model.

_Keywords_ photovoltaic forecasting $\cdot$ ensemble $\cdot$ cold-start
$\cdot$ automated design $\cdot$ online operation

##  1 Introduction

The Paris Climate Agreement targets net-zero emissions of the energy system by
2050 [1]. This target requires reducing primary energy demand, increasing
energy efficiency, and increasing the share of renewables in electricity
generation. The latter endangers grid stability, as the fluctuating and non-
controllable production behavior of renewables may put the frequency stability
under pressure or lead to power line congestion. Hence, a Smart Grid is
required that integrates communication, measurement, control, and automation
technology. This integration enables the Smart Grid to monitor local power
grid conditions in real-time and enables full utilization of the grid capacity
by balancing demand and supply [2]. The full utilization and balance of supply
and demand are realizable by using demand-side management and proactive supply
control. Yet, both rely on accurate demand and supply forecasts [3].

In this context, the automated design of such accurate forecasting models for
the decentral generation of PV power includes two challenges related to
missing information about an individual PV plant: First, the PV mounting
configuration characterized by inclination and azimuth angles is difficult to
obtain as both are often incompletely documented or roughly approximated.
Especially for PV plants with panels distributed on different roofs with
varying alignments, often only the total peak power rating of the PV plant is
known. This missing documentation limits the large-scale application of
forecasting methods that rely on information about the PV mounting
configuration (e.g. [4, 5]). Second, often historical data for training the
forecasting model are not available (cold-start problem). For example, for
newly built PV plants, nearly no data are available. Consequently, data-
intensive forecasting methods such as deep learning methods (e.g. [6, 7, 8])
are not directly applicable to new PV plants.

Therefore, the present paper addresses these two challenges by proposing a new
method for day-ahead PV forecasts called AutoPV. This method only requires the
new PV plant’s peak power rating, is applicable without any training data of
the new plant, and adapts itself during operation. Our proposed forecasting
method AutoPV is based on creating an ensemble of multiple forecasting models.
The models in the ensemble pool are pre-trained on historical data of
different PV plants of the same region with various mounting configurations
that are scaled with the corresponding peak power rating. Thus, the model pool
contains expert models for various PV mounting configurations. To tackle the
cold-start problem where no data of the PV plant are available, each model in
the ensemble contributes equally to the forecast. During the operation, new
data are collected and used to adapt the ensemble weights. More specifically,
the contribution of each model in the ensemble pool is adapted such that the
weighted sum optimally fits the new PV plant. Thereby, the unknown PV mounting
configuration is implicitly reflected in the ensemble weights. Also, PV plants
with panels distributed on different roofs with varying alignments can be
reflected proportionally in the weighting.

Regarding the taxonomy of automated forecasting pipelines described in [9],
our method covers all 5 sections, i.e., data pre-processing, feature
engineering, hyperparameter optimization, forecasting method selection, and
forecast ensembling. The organization of this workflow using a machine
learning pipeline, as well as automatic model design and selection along with
continuous adaptation during operation, corresponds to Automation level 3
[10]. For creating automated forecasting pipelines, generic frameworks such as
TPOT [11] exist, but they are not adapted to a specific domain and thus cannot
leverage prior knowledge on PV forecasting.

The remainder of this paper is organized as follows. We describe the
methodology of the proposed PV power generation forecasting method in Section
2, evaluate it in Section 3, discuss the results in Section 4 and give a
conclusion and an outlook in Section 5.

##  2 Design of the AutoPV method

As described in the introduction, unknown PV mounting configurations
characterized by inclination and azimuth angles are challenging for the design
of a PV forecasting model. Thus, the underlying idea of the proposed AutoPV
method is that each new PV mounting configuration can be described by the sum
of weighted elements from a sufficiently diverse pool of forecasting models of
the same region.

The proposed AutoPV method incorporates three steps: i) create the model pool,
ii) form the ensemble forecast by an optimally weighted sum of the scaled
forecasts, and iii) re-scale the ensemble forecast with the new PV plant’s
peak power rating. These three steps are detailed in the following and Figure
1 provides exemplary results of these three steps.

###  2.1 Creation of the ensemble pool models

Each model in the ensemble pool is equally designed. However, each model is
pre-trained on historical data of an individual PV plant. Thus, the ensemble
model pool covers a diverse set of PV mounting configurations. Each model is
based on a machine learning pipeline, including pre-processing and feature
extraction, an automatically designed regression estimator, and a set of rules
reflecting prior knowledge. First, we describe the machine learning pipeline
and afterward the automated regression estimator design.

#### Machine learning pipeline

For each model $n\ldots N$, we pre-process the training data by scaling the PV
power generation measurement according to the peak power rating
$P_{\text{n},n}$ of the corresponding PV plant

$\underaccent{\bar}{y}_{n}[k]=\frac{y_{n}[k]}{P_{\text{n},n}},$ (1)

where $k\in\mathbb{N}^{K}$ is the sample index. In contrast to the PV mounting
configurations, we assume that $P_{\text{n},n}$ is a parameter of the PV
plants that can be reliably determined. However, scaling with $P_{\text{n},n}$
does not lead to aligned PV power generation curves (see 1(a)). This is
because the mounting configuration of a PV plant influences the maximum
possible amount of generated power. For example, a PV plant oriented to the
south generates more than a PV plant oriented to the west. In addition, global
radiation’s seasonal and weather-dependent intensity also affects the distance
between the PV power output and the plant’s peak power rating.

As inputs, each regression estimator uses global radiation $\hat{G}$
considering cloud cover and air temperature $\hat{T}$ forecasts, and the
corresponding second-order polynomial and interaction features. Additionally,
we use sin-cos encoded cyclic features of the month and minute of the day to
represent seasonal information:

$\displaystyle x_{\text{s12}}[k]$
$\displaystyle=\sin(2\pi\cdot\text{month}[k]\cdot 12^{-1}),$ (2)
$\displaystyle x_{\text{c12}}[k]$
$\displaystyle=\cos(2\pi\cdot\text{month}[k]\cdot 12^{-1}),$ (3)
$\displaystyle x_{\text{s1440}}[k]$
$\displaystyle=\sin(2\pi\cdot\text{minute}[k]\cdot 1440^{-1}),$ (4)
$\displaystyle x_{\text{c1440}}[k]$
$\displaystyle=\cos(2\pi\cdot\text{minute}[k]\cdot 1440^{-1}).$ (5)

The periodicity of these trigonometric functions establishes similarities
between temporally related samples, while the sin-cos pair is necessary
because, otherwise, the encoding would be ambiguous at several points.111 We
do not consider historical data as lag features. The reason is that the models
in the ensemble pool are pre-trained on data of different PV plants.
Consequently, using historical data would require getting data from all used
PV plants in operation.

The resulting regression estimator for plant $n$ in the ensemble pool is a
function of

$\begin{split}\hat{\underaccent{\bar}{y}}_{n}[k]=f&\left(\hat{G}^{2}[k],\hat{G}[k],(\hat{G}\cdot\hat{T})[k],\hat{T}[k],\hat{T}^{2}[k],\right.\\\
&\left.\vphantom{\hat{G}^{2}[k],\hat{G}[k],(\hat{G}\cdot\hat{T})[k],\hat{T}[k],\hat{T}^{2}[k]}x_{\text{s12}}[k],x_{\text{c12}}[k],x_{\text{s1440}}[k],x_{\text{c1440}}[k],\boldsymbol{p}\right),\end{split}$
(6)

where the parameters $\boldsymbol{p}$ are determined by training the machine
learning pipeline on historical data.

To consider prior knowledge in forecasting PV power generation, we use two
rules. First, no PV power is generated if there is no solar radiation. Second,
negative PV power generation is not possible. Consequently, we drop the night
times from the training data and set the forecast to zero during these times.
Furthermore, the negative values in the forecast are set to zero.222 Slightly
negative values may occur in the first data points at sunrise and sunset.

Incorporating these two steps leads to

$\hat{\underaccent{\tilde}{y}}_{n}[k]=\begin{cases*}\hat{\underaccent{\bar}{y}}_{n}[k],&if
$\hat{G}[k]>0$ and $\hat{\underaccent{\bar}{y}}_{n}[k]>0$,\\\ 0&otherwise,\\\
\end{cases*}$ (7)

as the final step in the machine learning pipeline.

#### Automated regression estimator design

For automatically designing the regression estimator, we define a CASH
problem. In the CASH problem, we aim to minimize the machine learning
pipeline’s MSE

$\text{MSE}=\frac{1}{K}\sum_{k=1}^{K}\left(\hat{\underaccent{\tilde}{y}}_{n}[k]-\underaccent{\bar}{y}_{n}[k]\right)^{2},$
(8)

by selecting the optimal configuration, i.e., the optimal regression algorithm
and the corresponding optimal hyperparameters. The configuration space is
shown in Table 1, which is explored during CASH using Bayesian optimization.
Each configuration trial is trained on a training data set and assessed on a
hold-out validation data set.

#### Implementation

The machine learning pipeline is implemented using the Python package pyWATTS
[12] and the regression estimators are implemented using the Python package
scikit-learn [13]. For solving the CASH problem, we use the Python package Ray
Tune [14] with the hyperopt [15] search algorithm. The CASH is automatically
stopped if the MSE plateaus across trials, i.e., the MSE of the top 10 trials
have a standard deviation of less than 0.001 with a patience of 15 trials.

Table 1: The configuration space for the automated regression estimator design
using CASH in scikit-learn [13] naming convention.

Regression estimator Hyperparameter Value range Ridge alpha $[0.05,1]$
MLPRegressor activation $\\{\texttt{logistic, tanh, relu}\\}$
hidden_layer_sizes $\\{([10,100])$ $\phantom{\\{}([10,100],[10,100])$
$\phantom{\\{}([10,100],[10,100],[10,100])\\}$ GradientBoostingRegressor
learning_rate $[0.01,1]$ n_estimators $[10,300]$ max_depth $[1,10]$
RandomForestRegressor n_estimators $[10,300]$ max_depth $[1,10]$ SVR C
$[0.01,10]$ epsilon $[0.001,1]$

###  2.2 Optimal weighting of the ensemble pool models

The idea of creating an ensemble is to increase the robustness of data-driven
models by combining multiple forecasts from a pool of different models [16].
Apart from weighting each model in the pool equally (averaging), one may give
more weight to models from which we expect good performance and give less
weight to models from which we expect poor performance, i.e.,

$\hat{\underaccent{\tilde}{y}}[k]=\sum^{N}_{n=1}w_{n}\cdot\hat{\underaccent{\tilde}{y}}_{n}[k]$
(9)

with $w_{n}$ being the weight of the $n$-th model in the ensemble pool.

To find the weights of the ensemble, we distinguish the initialization and the
operation phase. In the initialization phase, historical power generation data
for the new PV plant are not available. Thus, we weight each model in the
ensemble pool equally. During the operation, new data are used to cyclically
adapt the ensemble weights in such a way that they optimally fit the new PV
plant (see 1(b)). For optimal weighting, we vary the weights of the ensemble
pool models $\boldsymbol{w}$ to minimize the MSE over the $K$ most recent
samples of the $P_{\text{n}}$-scaled data
$\underaccent{\tilde}{\boldsymbol{y}}$. With (9), this results in the
optimization problem

$\begin{split}\min_{\boldsymbol{w}}\frac{1}{K}\sum^{K}_{k=1}\left(\hat{\underaccent{\tilde}{y}}[k]-\underaccent{\bar}{y}[k]\right)^{2}\\\
\textrm{s.t.}\quad\boldsymbol{w}\in[0,1],\sum^{N}_{n=1}w_{n}=1,\end{split}$
(10)

which we solve using the least squares implementation of the Python package
SciPy [17], and normalize the weights afterward to hold the constraints.

Note that the cycle length $C$ of the adaption routine and the number of
considered samples $K$ are hyperparameters of the proposed AutoPV.

###  2.3 Transformation of the ensemble forecast to the new PV plant

The last step yields the final ensemble forecast, shown in 1(c)). To this end,
it re-scales the weighted average of the ensemble according to the peak power
rating $P_{\text{n,new}}$ of the new PV plant:

$\hat{y}[k]=\hat{\underaccent{\tilde}{y}}[k]\cdot P_{\text{n,new}}.$ (11)

(a) The result of the first step is the scaled ensemble pool model forecasts
$\hat{\underaccent{\tilde}{y}}_{n}$.

(b) The result of the second step is the scaled ensemble forecast
$\hat{\underaccent{\tilde}{y}}$.

(c) The result of the third step is the re-scaled ensemble forecast
$\hat{{y}}$. Additionally, this plot shows the measurement $y$ with cloud
overflight between 12:00 and 15:00. The ensemble forecast is based on a day-
ahead radiation forecast that also considers cloud cover but with lower
spatial and temporal resolution.

Figure 1: The three steps of the proposed PV ensemble forecasting method
exemplified for PV plant no. 11.

##  3 Evaluation

We evaluate the proposed AutoPV method on a real-world data set containing
three years (2018 – 2020) of quarter-hourly power generation measurement
($\mathrm{kW}\text{\,}\mathrm{h}$) from 11 PV plants located in southern
Germany. For better interpretability, we transform the energy generation time
series into the mean power generation time series

$\overline{P}[k]=\frac{\Delta E[k]}{t_{k}},$ (12)

where $\Delta E[k]$ is the energy generated in the sample period $t_{k}$
($1/$\mathrm{h}$$). Additionally, we use corresponding day-ahead weather
forecasting data.

We pre-train the ensemble pool models with the data from 2018 and 2019, and
assess the proposed AutoPV method with all samples of 2020. In pre-training,
the automated regression estimator design via CASH predominately creates a MLP
regressor with two or three hidden layers and the ReLU activation function.

In the following, we first evaluate the performance of AutoPV and then the
consistency of AutoPV’s ensemble optimization.

#### Performance evaluation

We evaluate the performance of AutoPV using a plant-wise leave-one-out
evaluation. That is, we assess the forecasting error of the proposed AutoPV
method on each PV plant while excluding this plant from the ensemble model
pool. Since the PV plants in the data set have different peak power ratings
and would differently influence the MAE, we use the nMAE as assessment metric:

$\text{nMAE}=\frac{\sum_{k=1}^{K}\left|\hat{y}[k]-y[k]\right|}{\sum_{k=1}^{K}{y[k]}}.$
(13)

In the evaluation, we start with equal ensemble weights and adapt them every
28 days based on the samples obtained in the most recent 28 days
($C=28,K=28\cdot 96$).

We compare the performance of the proposed AutoPV method with three methods.
The first method reflects an ideal situation in the sense that we assume that
there are two years of historical data available (2018, 2019) to train an
individual model (IM-HDA) for the considered PV plant. The second method is an
averaging ensemble, which reflects the initialization phase of AutoPV without
adapting the weights during operation. As a third method, we train an
individual model incrementally (IM-IT) every 28 days. More specifically, the
IM-IT uses all data of the testing data set (2020) that is obtained until the
$l$-th adaption ($C=28,K=l\cdot 28\cdot 96,l=[1,2,\ldots]$). The performance
comparison of IM-IT, Averaging, and AutoPV with the IM-HDA is unfair in terms
of training data because the IM-HDA is trained on data of the PV plant that is
unavailable to them.

In the performance evaluation, we make three observations. First, considering
the average assessment metric over all plants $\overline{\text{nMAE}}$, we see
that the proposed AutoPV method comes close to the ideal situation IM-HDA.
Second, regarding the cold-start problem, the proposed AutoPV method
outperforms the IM-IT (compare Table 2). Third, PV plant no. 7 has multiple
dips in September and October, as well as a complete shutdown of this plant in
May (see Figure 2), and the nMAE of all methods is comparatively high for this
plant.

Table 2: The nMAE of the proposed AutoPV method compared to the historical
data available model (IM-HDA), the averaging ensemble, and the incrementally
trained individual model (IM-IT) in the plant-wise leave-one-out evaluation.

IM-HDA IM-IT Averaging AutoPV Training data 2 years 1 month none none Adaption
cycle $C$ none 28d none 28d Adaption samples $K$ none $l\cdot 28\cdot 96$ none
$28\cdot 96$ $\text{nMAE}_{1\phantom{1}}$ 0.308 0.332 0.367 0.278
$\text{nMAE}_{2\phantom{1}}$ 0.298 0.321 0.302 0.299
$\text{nMAE}_{3\phantom{1}}$ 0.271 0.310 0.314 0.289
$\text{nMAE}_{4\phantom{1}}$ 0.316 0.349 0.385 0.291
$\text{nMAE}_{5\phantom{1}}$ 0.325 0.343 0.353 0.335
$\text{nMAE}_{6\phantom{1}}$ 0.327 0.322 0.299 0.289
$\text{nMAE}_{7\phantom{1}}$ 0.418 0.588 0.491 0.448
$\text{nMAE}_{8\phantom{1}}$ 0.280 0.310 0.280 0.277
$\text{nMAE}_{9\phantom{1}}$ 0.362 0.424 0.502 0.416 $\text{nMAE}_{10}$ 0.307
0.365 0.364 0.318 $\text{nMAE}_{11}$ 0.311 0.471 0.324 0.311
$\overline{\text{nMAE}}_{\phantom{1}\phantom{1}}$ 0.320 0.376 0.362 0.323
bolt: winner of fair comparison in terms of training data; bolt&italics:
winner of unfair comparison

Figure 2: Shutdowns and partial shutdowns of PV plant no. 7 lead to sudden
dips in the generated power.

#### Consistency evaluation

We evaluate the consistency of the AutoPV ensemble optimization as described
above, except that the ensemble model pool includes all PV plants, i.e., the
IM-HDA is available in the pool. Hence, a consistent ensemble optimization
should give high weight to the IM-HDA.

(a) PV plant no. 5 is rather similar compared to other plants.

(b) PV plant no. 8 is rather unique compared to other plants.

Figure 3: Weight development in the consistency evaluation.

Figure 3 shows the exemplary weight evolution of PV plant no. 5 and PV plant
no. 8. PV plant no. 5 represents a plant whose power generation curve is
rather similar to other plants, while PV plant no. 8 reflects a rather unique
power generation curve (compare in 1(a)).

In the consistency evaluation, we make three observations. First, after the
initialization phase, the weights change significantly. Second, the more
unique the power generation curve of a PV plant, the clearer the weighting
towards the IM-HDA (compare 3(a) and 3(b), where for PV plant no. 5 the weight
$w_{5}$ and for PV plant no. 8 the weight $w_{8}$ corresponds to the
respective IM-HDA). Third, the weighting is generally clearer towards the IM-
HDA in the summer than in the winter.

##  4 Discussion

This section discusses the results, limitations, and benefits of the proposed
AutoPV method.

With regard to the results, we discuss two aspects: First, we observe that the
proposed AutoPV method and the ideal IM-HDA perform almost similarly. Thus, we
conclude that any PV mounting configuration in the given dataset can be
represented by recombining the ensemble pool models’ forecasts. This
recombination via the ensemble weights also allows to represent PV plants with
panels distributed on different roofs with varying alignments. More
specifically, the AutoPV method is not limited to represent a single PV
mounting configuration. Instead, the mounting configurations of PV panels
distributed on different roofs can be reflected proportionally in the
weighting. Second, we observe that the proposed AutoPV method outperforms the
IM-IT. This might be explained by the fact that the pre-trained ensemble pool
models already reflect the entire seasonality, whereas the IM-IT learns it
with a delay. However, the forecasting accuracy of both is harmed by dips in
the generated PV power, as this sudden change is only captured with a delay by
the cyclic adaptation.

Regarding the limitations, we discuss the creation of the ensemble pool. For
representing a new PV plant by recombining the ensemble pool models’
forecasts, they must be diverse with respect to the PV mounting
configurations. To create a diverse ensemble model pool, we can use prior
knowledge or select a diverse set of curves after the $P_{\text{n}}$-scaling
(see 1(a)). However, we cannot guarantee that an ensemble pool of 10 models is
sufficiently diverse to represent all alignment combinations. This limitation
can be overcome by automatically designing an individual model for the new PV
plant after a full year and adding it to the ensemble pool.

The proposed AutoPV method is beneficial because it is universally applicable
and achieves promising results with only 10 representative models in the
ensemble pool. As scaling AutoPV to several hundred PV plants still requires
only the 10 ensemble pool models to be stored in the computing memory, and the
weight adaptation effort is very low, AutoPV is advantageous for Smart Grid
environments with limited computing capabilities. Moreover, the universality
of our approach allows us to use arbitrary complex models in the ensemble
pool.

##  5 Conclusion and outlook

Designing accurate PV power generation forecasting models for Smart Grid
operation includes two challenges: The first is the missing information about
the PV mounting configuration (inclination and azimuth angles). The second is
the limited availability of historical data to train the forecasting model
(cold-start problem). To address both challenges, we propose a new method for
day-ahead PV forecasts that only requires the new PV plant’s peak power
rating, that is applicable without any training data of the new plant, and
that adapts itself during operation (Automation level 3 [10]). The new AutoPV
method is an automated forecasting pipeline [9] based on an ensemble with a
pool of pre-trained models that represent various PV mounting configurations.
While initially, each model in the ensemble pool contributes equally to the
forecast, the contribution of each model is adapted during operation using the
least squares method such that the weighted sum optimally fits the new PV
plant. Hence, the AutoPV method is also applicable to PV plants with panels
distributed on different roofs with varying alignments, as these mounting
configurations can be reflected proportionally in the weighting.

The evaluation on real-world data shows that the proposed AutoPV method
outperforms an incrementally trained individual model. It achieves comparable
performance to the ideal situation of having two years of historical data
available to train an individual model. The evaluation also shows that for
reliable and automated online operation, it is necessary to consider PV power
generation dips caused by technical defects, scheduled maintenance, or
soiling. Therefore, the AutoPV method will be extended in future research to
include a drift detection method and consider the PV plant’s efficiency with
additional weight in the ensemble optimization.

## Acknowledgments

This project is funded by the Helmholtz Association’s Initiative and
Networking Fund through Helmholtz AI the Helmholtz Association under the
Program “Energy System Design”. Furthermore, the authors thank Stadtwerke
Karlsruhe Netzservice GmbH (Karlsruhe, Germany) for the data required for this
work.

## References

  * [1] UNFCCC  “Paris Agreement”  In _United Nations Treaty Collection_ Chapter XXVII 7. d, 2015
  * [2] Xi Fang, Satyajayant Misra, Guoliang Xue and Dejun Yang  “Smart Grid – The new and improved power grid: A survey”  In _IEEE Communications Surveys & Tutorials_ 14.4, 2012, pp. 944–980
  * [3] Lars Dannecker  “Energy time series forecasting”  Wiesbaden, Germany: Springer, 2015
  * [4] Oscar Perpiñán Lamigueiro  “solaR: Solar radiation and photovoltaic systems with R”  In _Journal of Statistical Software_ 50.9 American Statistical Association, 2012, pp. 1–32
  * [5] William F. Holmgren, Clifford W. Hansen and Mark A. Mikofski  “pvlib Python: A Python package for modeling solar energy systems”  In _Journal of Open Source Software_ 3.29 The Open Journal, 2018, pp. 884
  * [6] Mohamed Abdel-Basset, Hossam Hawash, Ripon K Chakrabortty and Michael Ryan  “PV-Net: An innovative deep learning approach for efficient forecasting of short-term photovoltaic energy production”  In _Journal of Cleaner Production_ 303 Elsevier, 2021, pp. 127037
  * [7] Muhammad Aslam, Seung-Jae Lee, Sang-Hee Khang and Sugwon Hong  “Two-stage attention over LSTM with Bayesian optimization for day-ahead solar power forecasting”  In _IEEE Access_ 9, 2021, pp. 107387–107398
  * [8] Kejun Wang, Xiaoxia Qi and Hongda Liu  “A comparison of day-ahead photovoltaic power forecasting models based on deep learning neural network”  In _Applied Energy_ 251.C, 2019, pp. 1–1
  * [9] Stefan Meisenbacher et al.  “Review of automated time series forecasting pipelines”  In _WIREs Data Mining and Knowledge Discovery_ 12.6, 2022, pp. e1475
  * [10] Stefan Meisenbacher et al.  “Concepts for automated machine learning in Smart Grid applications”  In _Proceedings - 31. Workshop Computational Intelligence : Berlin, 25. - 26. November 2021. Hrsg.: H. Schulte; F. Hoffmann; R. Mikut_ KIT Scientific Publishing, 2021, pp. 11–35
  * [11] Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz and Jason H. Moore  “Evaluation of a tree-based pipeline optimization tool for automating data science”  In _Proceedings of the Genetic and Evolutionary Computation Conference 2016_ , GECCO ’16  Denver, Colorado, USA: ACM, 2016, pp. 485–492
  * [12] Benedikt Heidrich et al.  “pyWATTS: Python workflow automation tool for time series”  In _arXiv:2106.10157_ , 2021
  * [13] F. Pedregosa et al.  “Scikit-learn: Machine learning in Python”  In _Journal of Machine Learning Research_ 12, 2011, pp. 2825–2830
  * [14] Richard Liaw et al.  “Tune: A research platform for distributed model selection and training”  In _arXiv:1807.05118_ , 2018
  * [15] James Bergstra, Daniel Yamins and David Cox  “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures”  In _International conference on machine learning_ , 2013, pp. 115–123  PMLR
  * [16] David Shaub  “Fast and accurate yearly time series forecasting with forecast combinations”  In _International Journal of Forecasting_ 36.1, 2020, pp. 116–120
  * [17] Pauli Virtanen et al.  “SciPy 1.0: Fundamental algorithms for scientific computing in Python”  In _Nature Methods_ 17, 2020, pp. 261–272


# 3D-aware Conditional Image Synthesis

Kangle Deng Gengshan Yang Deva Ramanan Jun-Yan Zhu
Carnegie Mellon University

###### Abstract

We propose pix2pix3D, a 3D-aware conditional generative model for controllable
photorealistic image synthesis. Given a 2D label map, such as a segmentation
or edge map, our model learns to synthesize a corresponding image from
different viewpoints. To enable explicit 3D user control, we extend
conditional generative models with neural radiance fields. Given widely-
available posed monocular image and label map pairs, our model learns to
assign a label to every 3D point in addition to color and density, which
enables it to render the image and pixel-aligned label map simultaneously.
Finally, we build an interactive system that allows users to edit the label
map from different viewpoints and generate outputs accordingly.

Figure 1: Given a 2D label map as input, such as a segmentation or edge map,
our model learns to predict high-quality 3D labels, geometry, and appearance,
which enables us to render both labels and RGB images from different
viewpoints. The inferred 3D labels further allow interactive editing of label
maps from any viewpoint, as shown in Figure 10.

##  1 Introduction

Content creation with generative models has witnessed tremendous progress in
recent years, enabling high-quality, user-controllable image and video
synthesis [20, 34, 19, 24]. In particular, image-to-image translation methods
[29, 86, 56] allow users to interactively create and manipulate a high-
resolution image given a 2D input label map. Unfortunately, existing image-to-
image translation methods operate purely in 2D, without explicit reasoning of
the underlying 3D structure of the content. As shown in Figure 1, we aim to
make conditional image synthesis 3D-aware, allowing not only 3D content
generation but also viewpoint manipulation and attribute editing (e.g., car
shape) in 3D.

Synthesizing 3D content conditioned on user input is challenging. For model
training, it is costly to obtain large-scale datasets with paired user inputs
and their desired 3D outputs. During test time, 3D content creation often
requires multi-view user inputs, as a user may want to specify the details of
3D objects using 2D interfaces from different viewpoints. However, these
inputs may not be 3D-consistent, providing conflicting signals for 3D content
creation.

To address the above challenges, we extend conditional generative models with
3D neural scene representations. To enable _cross-view_ editing, we
additionally encode semantic information in 3D, which can then be rendered as
2D label maps from different viewpoints. We learn the aforementioned 3D
representation using only 2D supervision in the form of image reconstruction
and adversarial losses. While the reconstruction loss ensures the alignment
between 2D user inputs and corresponding 3D content, our pixel-aligned
conditional discriminator encourages the appearance and labels to look
plausible while remaining pixel-aligned when rendered into novel viewpoints.
We also propose a cross-view consistency loss to enforce the latent codes to
be consistent from different viewpoints.

Figure 2: Overall pipeline. Given a 2D label map (e.g., segmentation map), a
random latent code $z$, and a camera pose $\hat{P}$ as inputs, our generator
renders the label map and image from viewpoint $\hat{P}$. Intuitively, the
input label map specifies the geometric structure, while the latent code
captures the appearance, such as hair color. We begin with an encoder that
encodes both the input label map and the latent code into style vectors
$\mathbf{w}^{+}$. We then use $\mathbf{w}^{+}$ to modulate our 3D
representation, which takes a spatial point $\mathbf{x}$ and outputs (1) color
$\mathbf{c}\in\mathbb{R}^{3}$, (2) density $\sigma$, (3) feature
$\phi\in\mathbb{R}^{l}$, and (4) label $\mathbf{s}\in\mathbb{R}^{c}$. We then
perform volumetric rendering and 2D upsampling to get the high-res label map
$\mathbf{\hat{I}}_{\mathbf{s}}^{+}$ and RGB Image
$\mathbf{\hat{I}}_{\mathbf{c}}^{+}$. For those rendered from ground-truth
poses, we compare them to ground-truth labels and images with an LPIPS loss
and label reconstruction loss. We apply a GAN loss on labels and images
rendered from both novel and original viewpoints.

We focus on 3D-aware semantic image synthesis on the CelebAMask-HQ [38], AFHQ-
cat[16], and shapenet-car[10] datasets. Our method works well for various 2D
user inputs, including segmentation maps and edge maps. Our method outperforms
several 2D and 3D baselines, such as Pix2NeRF variants [6], SofGAN [11], and
SEAN [89]. We further ablate the impact of various design choices and
demonstrate applications of our method, such as cross-view editing and
explicit user control over semantics and style. Please see our website for
more results and code.

##  2 Related Work

Neural Implicit Representation. Neural implicit fields, such as DeepSDF and
NeRFs [46, 54], model the appearance of objects and scenes with an implicitly
defined, continuous 3D representation parameterized by neural networks. They
have produced significant results for 3D reconstruction [90, 67] and novel
view synthesis applications [81, 43, 44, 39, 48] thanks to their compactness
and expressiveness. NeRF and its descendants aim to optimize a network for an
individual scene, given hundreds of images from multiple viewpoints. Recent
works further reduce the number of training views through learning network
initializations [79, 70, 13], leveraging auxiliary supervision [18, 30], or
imposing regularization terms [50]. Recently, explicit or hybrid
representations of radiance fields [48, 61, 12] have also shown promising
results regarding quality and speed. In our work, we use hybrid
representations for modeling both user inputs and outputs in 3D, focusing on
synthesizing novel images rather than reconstructing an existing scene. A
recent work Pix2NeRF [6] aims to translate a single image to a neural radiance
field, which allows single-image novel view synthesis. In contrast, we focus
on 3D-aware user-controlled content generation.

Conditional GANs. Generative adversarial networks (GANs) learn the
distribution of natural images by forcing the generated and real images to be
indistinguishable. They have demonstrated high-quality results on 2D image
synthesis and manipulation [20, 34, 35, 33, 85, 5, 59, 84, 72, 65, 3, 1].
Several methods adopt image-conditional GANs [29, 47] for user-guided image
synthesis and editing applications [86, 40, 74, 89, 56, 55, 26, 27, 62, 38] .
In contrast, we propose a 3D-aware generative model conditioned on 2D user
inputs that can render view-consistent images and enable interactive 3D
editing. Recently, SoFGAN [11] uses a 3D semantic map generator and a 2D
semantic-to-image generator to enable 3D-aware generation, but using 2D
generators does not ensure 3D consistency.

3D-aware Image Synthesis. Early data-driven 3D image editing systems can
achieve various 3D effects but often require a huge amount of manual effort
[37, 14]. Recent works have integrated the 3D structure into learning-based
image generation pipelines using various geometric representations, including
voxels [88, 22], voxelized 3D features [49], and 3D morphable models [78, 71].
However, many rely on external 3D data [88, 71, 78]. Recently, neural scene
representations have been integrated into GANs to enable 3D-aware image
synthesis [64, 51, 21, 8, 9, 52, 77, 53]. Intriguingly, these 3D-aware GANs
can learn 3D structures without any 3D supervision. For example, StyleNeRF
[21] and EG3D [8] learn to generate 3D representations by modulating either
NeRFs or explicit representations with latent style vectors. This allows them
to render high-resolution view-consistent images. Unlike the above methods, we
focus on conditional synthesis and interactive editing rather than random
sampling. Several works [28, 76, 42, 17] have explored sketch-based shape
generation but they do not allow realistic image synthesis.

Closely related to our work, Huang et al. [25] propose synthesizing novel
views conditional on a semantic map. Our work differs in three ways. First, we
can predict full 3D labels, geometry, and appearance, rather than only 2D
views, which enables cross-view editing. Second, our method can synthesize
images with a much wider baseline than Huang et al. [25]. Finally, our
learning algorithm does not require ground truth multi-view images of the same
scene. Two recent works, FENeRF [69] and 3DSGAN [80], also leverage semantic
labels for training 3D-aware GANs, but they do not support conditional inputs
and require additional efforts (e.g., GAN-inversion) to allow user editing.
Three concurrent works, IDE-3D [68], NeRFFaceEditing [31], and sem2nerf [15],
also explore the task of 3D-aware generation based on segmentation masks.
However, IDE-3D and sem2nerf only allow editing on a fixed view, and
NeRFFaceEditing focuses on real image editing rather than generation. All of
them do not include results for other input modalities. In contrast, we
present a general-purpose method that works well for diverse datasets and
input controls.

##  3 Method

Given a 2D label map ${\mathbf{I}}_{\mathbf{s}}$, such as a segmentation or
edge map, pix2pix3D generates a 3D-volumetric representation of geometry,
appearance, and labels that can be rendered from different viewpoints. Figure
2 provides an overview. We first introduce the formulation of our 3D
conditional generative model for 3D-aware image synthesis in Section 3.1.
Then, in Section 3.2, we discuss how to learn the model from color and label
map pairs $\\{\mathbf{I}_{\mathbf{c}},\mathbf{I}_{\mathbf{s}}\\}$ associated
with poses $\mathbf{P}$.

###  3.1 Conditional 3D Generative Models

Similar to EG3D [8], we adopt a hybrid representation for the density and
appearance of a scene and use style vectors to modulate the 3D generations. To
condition the 3D representations on 2D label map inputs, we introduce a
conditional encoder that maps a 2D label map into a latent style vector.
Additionally, pix2pix3D produces 3D labels that can be rendered from different
viewpoints, allowing for cross-view user editing.

Conditional Encoder. Given a 2D label map input ${\mathbf{I}}_{\mathbf{s}}$
and a random latent code sampled from the spherical Gaussian space
$\mathbf{z}\sim\mathcal{N}(0,I)$, our conditional encoder $E$ outputs a list
of style vectors $\mathbf{w}^{+}\in\mathbb{R}^{l\times 256}$,

$\mathbf{w}^{+}=E({\mathbf{I}}_{\mathbf{s}},\mathbf{z}),$

where $l=13$ is the number of layers to be modulated. Specifically, we encode
${\mathbf{I}}_{\mathbf{s}}$ into the first $7$ style vectors that represent
the global geometric information of the scene. We then feed the random latent
code $\mathbf{z}$ through a Multi-Layer Perceptron (MLP) mapping network to
obtain the rest of the style vectors that control the appearance.

Conditional 3D Representation. Our 3D representation is parameterized by tri-
planes followed by an $2$-layer MLP $f$ [8], which takes in a spatial point
$\mathbf{x}\in\mathbb{R}^{3}$ and returns 4 types of outputs: (1) color
$\mathbf{c}\in\mathbb{R}^{3}$, (2) density $\sigma\in\mathbb{R}^{+}$, (3)
feature $\phi\in\mathbb{R}^{64}$ for the purpose of 2D upsampling, and most
notably, (4) label $\mathbf{s}\in\mathbb{R}^{c}$, where $c$ is the number of
classes if ${\mathbf{I}}_{\mathbf{s}}$ is a segmentation map, otherwise 1 for
edge labels. We make the field conditional by modulating the generation of
tri-planes $F^{\text{tri}}$ with the style vectors $\mathbf{w}^{+}$. We also
remove the view dependence of the color following [8, 21]. Formally,

$(\mathbf{c},\mathbf{s},\sigma,\phi)=f(F^{\text{tri}}_{\mathbf{w}^{+}}(\mathbf{x})).$

Volume Rendering and Upsampling. We apply volumetric rendering to synthesize
color images [32, 46]. In addition, we render label maps, which are crucial
for enabling cross-view editing (Section 4.3) and improving rendering quality
(Table 1). Given a viewpoint $\hat{P}$ looking at the scene origin, we sample
$N$ points along the ray that emanates from a pixel location and query
density, color, labels, and feature information from our 3D representation.
Let ${\bf x_{i}}$ be the i-th sampled point along the ray $r$. Let ${\bf
c}_{i},{\bf s}_{i}$ and ${\bf\phi}_{i}$ be the color, labels, and the features
of ${\bf x_{i}}$. Similar to [69], The color, label map, and feature images
are computed as the weighted combination of queried values,

${\bf\hat{I}_{c}}(r)=\sum_{i=1}^{N}\tau_{i}{\bf
c}_{i},\;\;{\bf\hat{I}_{s}}(r)=\sum_{i=1}^{N}\tau_{i}{\bf
s}_{i},\;\;{\bf\hat{I}_{\phi}}(r)=\sum_{i=1}^{N}\tau_{i}{\bf{\phi}}_{i},$ (1)

where the transmittance ${\tau_{i}}$ is computed as the probability of a
photon traversing between the camera center and the i-th point given the
length of the i-th interval $\delta_{i}$,

${\tau_{i}}=\prod_{j=1}^{i}\exp{(-\sigma_{j}\delta_{j})}(1-\exp{(-\sigma_{i}\delta_{i}})).$

Similar to prior works [21, 52, 8], we approximate Equation 1 by 2D Upsampler
$U$ to reduce the computational cost. We render high-res $512\times 512$
images in two passes. In the first pass, we render low-res $64\times 64$
images ${\bf\hat{I}_{c}},{\bf\hat{I}_{s}},{\bf\hat{I}_{\phi}}$. Then a CNN up-
sampler $U$ is applied to obtain high-res images,

$\mathbf{\hat{I}}_{\mathbf{c}}^{+}=U({\bf\hat{I}_{c}},{\bf\hat{I}_{\phi}}),\qquad\mathbf{\hat{I}}_{\mathbf{s}}^{+}=U({\bf\hat{I}_{s}},{\bf\hat{I}_{\phi}}).$

###  3.2 Learning Objective

Learning conditional 3D representations from monocular images is challenging
due to its under-constrained nature. Given training data of associated images,
label maps, and camera poses predicted by an off-the-shelf model, we carefully
construct learning objectives, including reconstruction, adversarial, and
cross-view consistency losses. These objectives will be described below.

Reconstruction Loss. Given a ground-truth viewpoint ${\bf P}$ associated with
the color and label maps
$\\{{\mathbf{I}}_{\mathbf{c}},{\mathbf{I}}_{\mathbf{s}}\\}$, we render color
and label maps from ${\bf P}$ and compute reconstruction losses for both high-
res and low-res output. We use LPIPS [82] to compute the image reconstruction
loss $\mathcal{L}_{c}$ for color images. For label reconstruction loss
$\mathcal{L}_{s}$, we use the balanced cross-entropy loss for segmentation
maps or L2 Loss for edge maps,

$\mathcal{L}_{\text{recon}}=\lambda_{c}\mathcal{L}_{c}({\mathbf{I}}_{\mathbf{c}},\\{\mathbf{\hat{I}}_{\mathbf{c}},\mathbf{\hat{I}}_{\mathbf{c}}^{+}\\})+\lambda_{s}\mathcal{L}_{s}({\mathbf{I}}_{\mathbf{s}},\\{\mathbf{\hat{I}}_{\mathbf{s}},\mathbf{\hat{I}}_{\mathbf{s}}^{+}\\}),$

where $\lambda_{c}$ and $\lambda_{s}$ balance two terms.

Pixel-aligned Conditional Discriminator. The reconstruction loss alone fails
to synthesize detailed results from novel viewpoints. Therefore, we use an
adversarial loss [20] to enforce renderings to look realistic from random
viewpoints. Specifically, we have two discriminators $D_{\mathbf{c}}$ and
$D_{\mathbf{s}}$ for RGB images and label maps, respectively. $D_{\mathbf{c}}$
is a widely-used GAN loss that takes real and fake images as input, while the
pixel-aligned conditional discriminator $D_{\mathbf{s}}$ concatenates color
images and label maps as input, which encourages pixel alignment between color
images and label maps. Notably, in $D_{\mathbf{s}}$, we stop the gradients for
the color images to prevent a potential quality downgrade. We also feed the
rendered low-res images to prevent the upsampler from hallucinating details,
inconsistent with the low-res output. The adversarial loss can be written as
follows.

$\mathcal{L}_{\text{GAN}}=\lambda_{D_{\bf
c}}\mathcal{L}_{D_{\mathbf{c}}}(\mathbf{\hat{I}}_{\mathbf{c}}^{+},\mathbf{\hat{I}}_{\mathbf{c}})+\lambda_{D_{\bf
s}}\mathcal{L}_{D_{\mathbf{s}}}(\mathbf{\hat{I}}_{\mathbf{c}}^{+},\mathbf{\hat{I}}_{\mathbf{c}},\mathbf{\hat{I}}_{\mathbf{s}}^{+},\mathbf{\hat{I}}_{\mathbf{s}}).$

where $\lambda_{D_{\bf c}}$ and $\lambda_{D_{\bf s}}$ balance two terms. To
stabilize the GAN training, we adopt the R1 regularization loss [45].

Cross-view Consistency Loss. We observe that inputting label maps of the same
object from different viewpoints will sometimes result in different 3D shapes.
Therefore we add a cross-view consistency loss to regularize the training, as
illustrated in Figure 3. Given an input label map ${\mathbf{I}}_{\mathbf{s}}$
and its associated pose ${\bf P}$, we generate the label map
$\mathbf{\hat{I}}_{\mathbf{s}}^{\prime}$ from a different viewpoint ${\bf
P}^{\prime}$, and render the label map
$\mathbf{\hat{I}}_{\mathbf{s}}^{\prime\prime}$ back to the pose ${\bf P}$
using $\mathbf{\hat{I}}_{\mathbf{s}}^{\prime}$ as input. We add a
reconstruction loss between $\mathbf{\hat{I}}_{\mathbf{s}}^{\prime\prime}$ and
$\mathbf{\hat{I}}_{\mathbf{s}}$:

$\mathcal{L}_{\text{CVC}}=\lambda_{\text{CVC}}\mathcal{L}_{s}(\mathbf{\hat{I}}_{\mathbf{s}}^{\prime\prime},\mathbf{\hat{I}}_{\mathbf{s}}),$

where $\mathcal{L}_{s}$ denotes the reconstruction loss in the label space,
and $\lambda_{\text{CVC}}$ weights the loss term. This loss is crucial for
reducing error accumulation during cross-view editing.

Figure 3: Cross-View Consistency Loss. Given an input label map
${\mathbf{I}}_{\mathbf{s}}$ and its associated pose ${\bf P}$, we first infer
the geometry latent code ${{\bf w_{g}}}$. From ${{\bf w_{g}}}$, we can
generate a label map ${\mathbf{\hat{I}}}_{\mathbf{s}}$ from the same pose
${\bf P}$, and $\mathbf{\hat{I}}_{\mathbf{s}}^{\prime}$ from a random pose
${\bf P^{\prime}}$. Next, we infer ${\bf w_{g}^{\prime}}$ from the novel view
$\mathbf{\hat{I}}_{\mathbf{s}}^{\prime}$, and render it back to the original
pose ${\bf P}$ to obtain $\mathbf{\hat{I}}_{\mathbf{s}}^{\prime\prime}$.
Finally, we add a reconstruction loss:
$\mathcal{L}_{\text{CVC}}=\lambda_{\text{CVC}}\mathcal{L}_{s}(\mathbf{\hat{I}}_{\mathbf{s}}^{\prime\prime},\mathbf{\hat{I}}_{\mathbf{s}})$.

Optimization. Our final learning objective is written as follows:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{recon}}+\mathcal{L}_{\text{GAN}}+\mathcal{L}_{\text{CVC}}.$

At every iteration, we determine whether to use a ground-truth pose or sample
a random one with a probability of $p$. We use the reconstruction loss and GAN
loss for ground-truth poses, while for random poses, we only use the GAN loss.
We provide the hyper-parameters and more implementation details in Appendix B.

Figure 4:  Qualitative Comparison with Pix2NeRF[6], SoFGAN[11], and SEAN[89]
on CelebAMask dataset for seg2face task. SEAN fails in multi-view synthesis,
while SoFGAN suffers from multi-view inconsistency (e.g., face identity
changes across viewpoints). Our method renders high-quality images while
maintaining multi-view consistency. Please check our website for more
examples.

Seg2Face | Quality | Alignment |
---|---|---|---
|  | SG |  |  | FVV
CelebAMask[38] |  FID $\downarrow$ |  KID $\downarrow$ |  Diversity $\uparrow$ |  mIoU $\uparrow$ |  acc $\uparrow$ |  Identity $\downarrow$
SEAN [89] | 32.74 | 0.018 | 0.29 | 0.52 | 0.85 | N/A
SoFGAN [11] | 23.34 | 0.012 | 0.33 | 0.53 | 0.89 | 0.58
Pix2NeRF [6] | 54.23 | 0.042 | 0.16 | 0.36 | 0.65 | 0.44
pix2pix3D (Ours) |  |  |  |  |  |
w/o 3D Labels | 12.96 | 0.005 | 0.30 | N/A (0.43) | N/A (0.81) | 0.38
w/o CVC | 11.62 | 0.004 | 0.30 | 0.50 (0.50) | 0.87 (0.85) | 0.42
full model | 11.54 | 0.003 | 0.28 | 0.51 (0.52) |  0.90 (0.88) | 0.36
full model† | 11.13 | 0.003 | 0.29 | 0.51 (0.50) |  0.90 (0.87) | 0.36
Table 1: Seg2face Evaluation. Our metrics include image quality (FID, KID, SG Diversity), alignment (mIoU and acc against GT label maps), and multi-view consistency (FVV Identity). Single-generation diversity (SG Diversity) is obtained by computing the LPIPS metric between randomly generated pairs given a single conditional input. To evaluate alignment, we compare the generated label maps against the ground truth in terms of mIoU and pixel accuracy (acc). Alternatively, given a generated image, one could estimate label maps via a face parser, and compare those against the ground truth (numbers in parentheses). We include SEAN [89] and SoFGAN [11] as baselines, and modify Pix2NeRF [6] to take conditional input. Our method achieves the best quality, alignment ACC, and FVV Identity while being competitive on SG Diversity. SoFGAN tends to have better alignment but worse 3D consistency. We also ablate our method w.r.t the 3D labels and the cross-view consistency (CVC) loss. Our 3D labels are crucial for alignment, while the CVC loss improves multi-view consistency. Using pretrained models from EG3D ($\dagger$) also improves the performance.  Edge2Car | Quality | Alignment
---|---|---
FID $\downarrow$ |  KID $\downarrow$ |  SG Diversity $\uparrow$ |  AP $\uparrow$
Pix2NeRF [6] | 23.42 | 0.014 | 0.06 | 0.28
pix2pix3D (Ours) |  |  |  |
w/o 3D Labels | 10.73 | 0.005 | 0.12 | 0.45 (0.42)
w/o CVC | 9.42 | 0.004 | 0.13 | 0.61 (0.59)
full model | 8.31 | 0.004 | 0.13 |  0.63 (0.59)
Table 2: Edge2car Evaluation. We compare our method with Pix2NeRF[6] on
edge2car using the shapenet-car[10] dataset. Similar to Table 1, we evaluate
FID, KID, and SG Diversity for image quality. We also evaluate the alignment
with the input edge map using AP. Similarly, we can either run informative
drawing[7] on generated images to obtain edge maps (numbers in parentheses) or
directly use generated edge maps to calculate the metrics. We achieve better
image quality and alignment than Pix2NeRF. We also find that using 3D labels
and cross-view consistency loss is helpful regarding FID and AP metrics.

Figure 5: Qualitative ablation on seg2face and seg2cat. We ablate our method
by removing the branch that renders label maps (w/o 3D Labels). Our results
better align with input labels (e.g., hairlines and the cat’s ear).

Figure 6: Results on edge2cat. Our model is trained on AFHQ-cat[16] with edges extracted by pidinet[66]. Figure 7: Qualitative comparisons on edge2car. pix2pix3D (Ours) and Pix2NeRF[6] are trained on shapenet-car[10], and pix2pix3D achieves better quality and alignment than Pix2NeRF. Seg2Cat | Quality | Alignment
---|---|---
AFHQ-cat[34] |  FID $\downarrow$ |  KID $\downarrow$ |  SG Diversity $\uparrow$ |  mIoU $\uparrow$ |  acc $\uparrow$
Pix2NeRF [6] | 43.92 | 0.081 | 0.15 | 0.27 | 0.58
Ours |  |  |  |  |
w/o 3D Labels | 10.41 | 0.004 | 0.26 | N/A (0.49) | N/A (0.69)
w/o CVC | 9.64 | 0.004 | 0.26 |  0.66 (0.63) | 0.76 (0.73)
Full Model | 8.62 | 0.003 | 0.27 |  0.66 (0.62) |  0.78 (0.73)
Table 3: Seg2cat Evaluation. We compare our method with Pix2NeRF [6] on
Seg2Cat using AFHQ-cat dataset[16], with segmentation obtained by clustering
DINO features [2]. Similar to Table 1, we evaluate the image quality and
alignment. Ours performs better in all metrics. Figure 8: Semantic Mesh. We
show semantic meshes of human and cat faces from marching cubes colored by 3D
labels. Figure 9:  We study the effect of random pose sampling probability $p$
during training. Without random poses ($p=0$), the model achieves the best
alignment with input semantic maps, with reduced image quality. In contrast,
only using random poses ($p=1$) achieves the best image quality, while results
fail to align with input maps. We find $p=0.5$ balances the image quality and
input alignment.

Figure 10: Cross-view Editing of Edge2Car. Our 3D editing system allows users
to edit label maps from any viewpoint instead of only the input view.
Importantly, our feed-forward encoder allows fast inference of the latent code
without GAN-inversion. Typically, a single forward pass of rendering takes
only 40 ms on a single RTX A5000, which enables interactive editing. Please
check our demo video on our website.

##  4 Experiment

We first introduce the datasets and evaluation metrics. Then we compare our
method with the baselines. Finally, we demonstrate cross-view editing and
multi-modal synthesis applications enabled by our method.

Datasets. We consider four tasks: seg2face, seg2cat, edge2cat, and edge2car in
our experiments. For seg2face, we use CelebAMask-HQ [38] for evaluation.
CelebAMask-HQ contains 30,000 high-resolution face images from CelebA [41],
and each image has a facial part segmentation mask and a predicted pose. The
segmentation masks contain 19 classes, including skin, eyebrows, ears, mouth,
lip, etc. The pose associated with each image segmentation is predicted by
HopeNet [60]. We split the CelebAMask-HQ dataset into a training set of
24,183, a validation set of 2,993, and a test set of 2,824, following the
original work [38]. For seg2cat and edge2cat, we use AFHQ-cat [16], which
contains 5,065 images at $512\times$ resolution. We estimate the viewpoints
using unsup3d [75]. We extract the edges using pidinet [66] and obtain
segmentation by clustering DINO features [2] into 6 classes. For edge2car, we
use 3D models from shapenet-car [10] and render 500,000 images at $128\times$
resolution for training, and 30,000 for evaluation. We extract the edges using
informative drawing [7]. We train our model at $512\times$ resolution except
for $128\times$ in the edge2car task.

Running Time. For training the model at $512\times$ resolution, it takes about
three days on eight RTX 3090 GPUs. But we can significantly reduce the
training time to 4 hours if we initialize parts of our model with pretrained
weights from EG3D[8]. During inference, our model takes 10 ms to obtain the
style vector, and another 30 ms to render the final image and the label map on
a single RTX A5000. The low latency (25 FPS) allows for interactive user
editing.

###  4.1 Evaluation metrics

We evaluate the models from two aspects: 1) the image quality regarding
fidelity and diversity, and 2) the alignment between input label maps and
generated outputs.

Quality Metrics. Following prior works [57, 21], we use the clean-fid library
[58] to compute Fréchet Inception Distance (FID) [23] and Kernel Inception
Distance (KID) [4] to measure the distribution distance between synthesized
results and real images. We also evaluate the single-generation diversity (SG
Diversity) by calculating the LPIPS metric between randomly generated pairs
given a single input following prior works [87, 11]. For FID and KID, we
generate 10 images per label map in the test set using randomly sampled $z$.
We compare our generated images with the whole dataset, including training and
test images.

Alignment Metrics. We evaluate models on the test set using mean Intersection-
over-Union (mIoU) and pixel accuracy (acc) for segmentation maps following
existing works [57, 63], and average precision (AP) for edge maps. For those
models that render label maps as output, we directly compare them with ground-
truth labels. Otherwise, we first predict the label maps from the output RGB
images using off-the-shelf networks [38, 66], and then compare the prediction
with the ground truth. The metrics regarding such predicted semantic maps are
reported within brackets in Table 1 and Table 2.

For seg2face, we evaluate the preservation of facial identity from different
viewpoints (FVV Identity) by calculating their distances with the dlib face
recognition algorithm***https://github.com/ageitgey/face_recognition.

###  4.2 Baseline comparison

Baselines. Since there are no prior works on conditional 3D-aware image
synthesis, we make minimum modifications to Pix2NeRF [6] to be conditional on
label maps instead of images. For a thorough comparison, we introduce several
baselines: SEAN [89] and SoFGAN [11]. 2D baselines like SEAN [89] cannot
generate multi-view images by design (N/A for FVV Identity), while SoFGAN [11]
uses an unconditional 3D semantic map generator before the 2D generator so we
can evaluate FVV Identity for that.

Results. Figure 4 shows the qualitative comparison for seg2face and Table 1
reports the evaluation results. SoFGAN[11] tends to produce results with
slightly better alignment but worse 3D consistency for its 2D RGB generator.
Our method achieves the best quality, alignment acc, and FVV Identity while
being competitive with 2D baselines on SG diversity. Figure 5 shows the
qualitative ablation on seg2face and seg2cat. Table 5 reports the metrics for
seg2cat. Figure 6 shows the example results for edge2cat. Figure 7 shows the
qualitative comparison for edge2car and Table 2 reports the metrics. Our
method achieves the best image quality and alignment. Figure 8 shows semantic
meshes of human and cat faces, extracted by marching cubes and colored by our
learned 3D labels. We provide more evaluation results in Appendix A.

Figure 11: Multi-modal Synthesis. The leftmost column is the input
segmentation map. We use the same segmentation map for each row. We generate
multi-modal results by randomly sampling an appearance style for each column.

Figure 12: Interpolation. In each $5\times 5$ grid, the images at the top left
and bottom right are generated from the input maps next to them. Each row
interpolates two images in label space, while each column interpolates the
appearance. For camera poses, we interpolate the pitch along the row and the
yaw along the column.

Ablation Study. We compare our full method to several variants. Specifically,
(1) w/o 3D Labels, we remove the branch of rendering label maps from our
method, and (2) w/o CVC, we remove the cross-view consistency loss. From Table
1, Table 2, and Figure 5, rendering label maps is crucial for the alignment
with the input. We posit that the joint learning of appearance, geometry, and
label information poses strong constraints on correspondence between the input
label maps and the 3D representation. Thus our method can synthesize images
pixel-aligned with the inputs. Our CVC loss helps preserve the facial identity
from different viewpoints.

Analysis on random sampling of poses. We study the effect of the different
probabilities of sampling random poses during training, as shown in Figure 9.
When sampling no random poses ($p=0$), the model best aligns with input label
maps with suboptimal image quality. Conversely, only sampling random poses
($p=1$) gives the best image quality but suffers huge misalignment with input
label maps. We find $p=0.5$ achieves the balance between the image quality and
the alignment with the input.

###  4.3 Applications

Cross-view Editing. As shown in Figure 10, our 3D editing system allows users
to generate and edit label maps from any viewpoint instead of only the input
view. The edited label map is further fed into the conditional encoder to
update the 3D representation. Unlike GAN inversion [85], our feed-forward
conditional encoder allows fast inference of the latent code. Thus, a single
forward pass of our full model takes only 40 ms on a single RTX A5000.

Multi-modal synthesis and interpolation. Like other style-based generative
models [34, 21, 36, 8], our method can disentangle the geometry and appearance
information. Specifically, the input label map captures the geometry
information while the randomly sampled latent code controls the appearance. We
show style manipulation results in Figure 11. We can also interpolate both the
geometry styles and the appearance styles (Figure 12). These results show the
clear disentanglement of our 3D representation.

##  5 Discussion

We have introduced pix2pix3D, a 3D-aware conditional generative model for
controllable image synthesis. Given a 2D label map, our model allows users to
render images given any viewpoint. Our model augments the neural field with 3D
labels, assigning label, color, and density to every 3D point, allowing for
the simultaneous rendering of the image and a pixel-aligned label map. The
learned 3D labels further enable interactive 3D cross-view editing. We discuss
the broader impact and limitations in the appendix.

Acknowledgments. We thank Sheng-Yu Wang, Nupur Kumari, Gaurav Parmer, Ruihan
Gao, Muyang Li, George Cazenavette, Andrew Song, Zhipeng Bao, Tamaki Kojima,
Krishna Wadhwani, Takuya Narihira, and Tatsuo Fujiwara for their discussion
and help. We are grateful for the support from Sony Corporation, Singapore
DSTA, and the CMU Argo AI Center for Autonomous Vehicle Research.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.  Deep vit features as dense visual descriptors.  ECCVW What is Motion For?, 2022.
  * [3] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba.  Semantic photo manipulation with a generative image prior.  In ACM SIGGRAPH, 2019.
  * [4] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  In International Conference on Learning Representations (ICLR), 2018\.
  * [5] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In International Conference on Learning Representations (ICLR), 2019\.
  * [6] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool.  Pix2nerf: Unsupervised conditional $\pi$-gan for single image to neural radiance fields translation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [7] Caroline Chan, Frédo Durand, and Phillip Isola.  Learning to generate line drawings that convey geometry and semantics.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [8] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3D generative adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [9] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [10] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [11] Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, and Jingyi Yu.  Sofgan: A portrait image generator with dynamic styling.  In ACM SIGGRAPH, 2021.
  * [12] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In European Conference on Computer Vision (ECCV), 2022.
  * [13] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [14] Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or.  3-sweep: Extracting editable objects from a single photo.  ACM Transactions on Graphics (TOG), 32(6):1–10, 2013.
  * [15] Yuedong Chen, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  Sem2nerf: Converting single-view semantic masks to neural radiance fields.  In European Conference on Computer Vision (ECCV), 2022.
  * [16] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [17] Johanna Delanoy, Adrien Bousseau, Mathieu Aubry, Phillip Isola, and Alexei A Efros.  What you sketch is what you get: 3d sketching using multi-view deep volumetric prediction.  In ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.
  * [18] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised NeRF: Fewer views and faster training for free.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [19] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, 2014.
  * [21] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d aware generator for high-resolution image synthesis.  In International Conference on Learning Representations (ICLR), 2022\.
  * [22] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel.  Escaping plato’s cave: 3d shape from adversarial rendering.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [24] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [25] Hsin-Ping Huang, Hung-Yu Tseng, Hsin-Ying Lee, and Jia-Bin Huang.  Semantic view synthesis.  In European Conference on Computer Vision (ECCV), 2020.
  * [26] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In European Conference on Computer Vision (ECCV), 2018.
  * [27] Zeng Huang, Tianye Li, Weikai Chen, Yajie Zhao, Jun Xing, Chloe Legendre, Linjie Luo, Chongyang Ma, and Hao Li.  Deep volumetric video from very sparse multi-view performance capture.  In European Conference on Computer Vision (ECCV), pages 351–369, 2018.
  * [28] Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka.  Teddy: a sketching interface for 3d freeform design.  In ACM SIGGRAPH, 1999.
  * [29] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [30] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [31] Kaiwen Jiang, Shu-Yu Chen, Feng-Lin Liu, Hongbo Fu, and Lin Gao.  Nerffaceediting: Disentangled face editing in neural radiance fields.  In ACM SIGGRAPH Asia, 2022.
  * [32] James T Kajiya and Brian P Von Herzen.  Ray tracing volume densities.  ACM SIGGRAPH, 18(3):165–174, 1984.
  * [33] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [34] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [35] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [36] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [37] Natasha Kholgade, Tomas Simon, Alexei Efros, and Yaser Sheikh.  3d object manipulation in a single photograph using stock 3d models.  ACM Transactions on Graphics (TOG), 33(4):1–12, 2014.
  * [38] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [39] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey.  Barf: Bundle-adjusting neural radiance fields.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [40] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  Advances in neural information processing systems, 30, 2017.
  * [41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), 2015.
  * [42] Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang.  3d shape reconstruction from sketches via multi-view convolutional networks.  In 2017 International Conference on 3D Vision (3DV). IEEE, 2017\.
  * [43] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [44] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu.  Gnerf: Gan-based neural radiance field without posed camera.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [45] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  In International Conference on Machine Learning (ICML), 2018.
  * [46] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European Conference on Computer Vision (ECCV), 2020.
  * [47] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [48] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  In ACM SIGGRAPH, 2022.
  * [49] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [50] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [51] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [52] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.  Stylesdf: High-resolution 3d-consistent image and geometry generation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [53] Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai.  A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [54] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [55] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu.  Contrastive learning for unpaired image-to-image translation.  In European Conference on Computer Vision (ECCV), 2020.
  * [56] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [57] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [58] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On aliased resizing and surprising subtleties in gan evaluation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [59] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [60] Nataniel Ruiz, Eunji Chong, and James M. Rehg.  Fine-grained head pose estimation without keypoints.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2018.
  * [61] Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [62] Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva.  You only need adversarial supervision for semantic image synthesis.  In International Conference on Learning Representations (ICLR), 2020\.
  * [63] Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva.  You only need adversarial supervision for semantic image synthesis.  In International Conference on Learning Representations (ICLR), 2021\.
  * [64] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [65] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [66] Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikainen, and Li Liu.  Pixel difference networks for efficient edge detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [67] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison.  iMAP: Implicit mapping and positioning in real-time.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [68] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu.  Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.  In ACM Transactions on Graphics (TOG), 2022.
  * [69] Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, and Jue Wang.  Fenerf: Face editing in neural radiance fields.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [70] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng.  Learned initializations for optimizing coordinate-based neural representations.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [71] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt.  Stylerig: Rigging stylegan for 3d control over portrait images.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [72] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  In ACM Transactions on Graphics (TOG), 2021.
  * [73] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros.  Cnn-generated images are surprisingly easy to spot…for now.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [74] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [75] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [76] Xiaohua Xie, Kai Xu, Niloy J Mitra, Daniel Cohen-Or, Wenyong Gong, Qi Su, and Baoquan Chen.  Sketch-to-design: Context-based part assembly.  In Computer Graphics Forum, volume 32, pages 233–245. Wiley Online Library, 2013.
  * [77] Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou.  3d-aware image synthesis via learning structural and textural representations.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [78] Shunyu Yao, Tzu Ming Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum.  3d-aware scene manipulation via inverse graphics.  In Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * [79] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  Pixelnerf: Neural radiance fields from one or few images.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [80] Jichao Zhang, Enver Sangineto, Hao Tang, Aliaksandr Siarohin, Zhun Zhong, Nicu Sebe, and Wei Wang.  3d-aware semantic-guided generative model for human synthesis.  In European Conference on Computer Vision (ECCV), 2022.
  * [81] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv preprint arXiv:2010.07492, 2020.
  * [82] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [83] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [84] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In European Conference on Computer Vision (ECCV), 2020.
  * [85] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros.  Generative visual manipulation on the natural image manifold.  In European Conference on Computer Vision (ECCV), 2016.
  * [86] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [87] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman.  Toward multimodal image-to-image translation.  Advances in neural information processing systems, 30, 2017.
  * [88] Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman.  Visual object networks: Image generation with disentangled 3d representations.  In Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * [89] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka.  Sean: Image synthesis with semantic region-adaptive normalization.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [90] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys.  Nice-slam: Neural implicit scalable encoding for slam.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

## Appendix

Figure 13: Cross-view Editing of Seg2cat. The 3D representation can be edited
from a viewpoint different than the input seg map.

We include additional experimental results, implementation details, and the
societal impact of our work. Please also view our webpage for our interactive
editing demo video and additional visual results.

##  Appendix A Additional Experiments

Cross-view Editing of Seg2cat. In addition to the edge2car editing example in
the main paper, we showcase the editability of segmentation maps in Figure 13.
Note that the edited segmentation map does not have to be from the same
viewpoint as the input segmentation map.

Ablation study on discriminator design.  In the main paper, we introduce our
Pixel-aligned Conditional Discriminator that concatenates RGB images and label
maps as input. To verify the effectiveness of our discriminator design, we
introduce three ablation experiments in Table 4. We find that our image
discriminator helps improve the image quality, while our pixel-aligned
conditional discriminator is crucial for the alignment.

CelebA-Mask | Quality | Alignment
---|---|---
FID $\downarrow$ |  KID $\downarrow$ |  mIoU $\uparrow$ |  acc $\uparrow$
Ours | 11.54 | 0.003 |  0.51 (0.52) |  0.90 (0.88)
w/o Image D | 15.32 | 0.006 |  0.51 (0.52) | 0.89 (0.85)
w/o Conditional D | 12.02 | 0.004 | 0.37 (0.47) | 0.82 (0.80)
w/o Pixel-Align D | 11.94 | 0.003 | 0.41 (0.40) | 0.82 (0.81)
Table 4: Ablation Study on Discriminator Design. To verify the effectiveness
of our discriminator design, we introduce three ablation experiments: (1)w/o
Image D, we remove the image discriminator and only keep the conditional
discriminator that accepts the concatenation of image and segmentation maps;
(2)w/o Conditional D, we remove the conditional discriminator and only keeps
the image discriminator; (3)w/o Pixel-Align D, we keep both discriminators,
but the conditional discriminator no longer concatenates color images as part
of the input. Our image discriminator improves the image quality, while our
pixel-aligned conditional discriminator ensures alignment.

Evaluation on Seg2Car. We evaluate our method on an additional non-face
dataset Seg2Car, where we get the segmentation model from DatasetGAN[83]. We
show the visual and evaluation results in the Figure 14 and Table 5. We find
our method outperforms Pix2NeRF[6].

Figure 14: Visual Results of Seg2Car. Seg2Car | Quality | Alignment
---|---|---
Shapnet-car |  FID $\downarrow$ |  KID $\downarrow$ |  SG Diversity $\uparrow$ |  mIoU $\uparrow$ |  acc $\uparrow$
Pix2NeRF | 25.86 | 0.018 | 0.08 | 0.24 | 0.59
Ours | 9.35 | 0.004 | 0.14 | 0.58 | 0.88
Table 5: Seg2car Evaluation. We compare our method with Pix2NeRF[6]. Ours
performs better in all metrics.

Figure 15 compares our method with SoFGAN [11] regrading multi-view
consistency. We also show our method’s capability of correcting errors in the
user input in Figure 16.

Figure 15: Multi-view Consistency. We compare our method with SoFGAN [11]
regarding multi-view consistency. Although SoFGAN can generate images from
different viewpoints, their method shifts the face identity across viewpoints.
In contrast, our method better preserves the identity.  Figure 16: Our model
projects the user input onto the learned manifold. Even if the user input
contains errors, our model is able to fix them, e.g., completing the missing
eyes and eyebrows.

##  Appendix B Implementation Details

Class-balanced cross-entropy. In Section 3.2 of the main text, we mentioned
using class-balanced cross-entropy loss for reconstructing 2D segmentation
maps. Specifically,

$\mathcal{L}_{s}(\hat{\mathbf{I}}_{\mathbf{s}},\mathbf{I}_{\mathbf{s}}^{+})=\mathbb{E}_{n}-\sum_{c=1}^{C}w_{c}\log\frac{\exp(x_{n,c})}{\sum_{i=1}^{C}\exp(x_{n,i})}y_{n,c}.$

where $x_{n,c}$ is the semantic logits of class $c$ at location $n$ of
$\mathbf{I}_{\mathbf{s}}^{+}$, $y_{n,c}$ is the ground-truth probability of
class $c$ at location $n$ of $\hat{\mathbf{I}}_{\mathbf{s}}$, and $w_{c}$ is
the weight of each class $c$.

In our case, 2D segmentation maps are imbalanced as skin and hair cover a lot
more areas than the other classes. So we calculate $w_{c}$ based on the
inverse frequency of the classes in the training set,

$w_{c}=\sqrt{\frac{\text{\\# of pixels with class $c$}}{\text{\\# of all
pixels}}}.$

Regularization. As mentioned in the main text, we use non-saturating loss [20]
and R1 Regularization [45] for GAN training following [21, 36]. Specifically,

$\displaystyle\mathcal{L}_{\text{GAN}}(G,D)=$
$\displaystyle\mathbb{E}[f(D(G(z,\mathbf{\hat{I}_{s}}))]$ $\displaystyle+$
$\displaystyle\mathbb{E}[f(-D(\mathbf{\hat{I}_{c}}))+\lambda||\nabla
D(\mathbf{\hat{I}_{c}})||^{2}],$

where $G$ is the generator, $D$ is the discriminator,
$f(u)=-\log(1+\exp(-u))$, and $\lambda=0.5$.

Hyper-parameters. $\lambda_{c}=1,\lambda_{s}=5$ for edge maps, $\lambda_{s}=1$
for segmentation maps, $\lambda_{D_{\bf c}}=1$, $\lambda_{D_{\bf s}}=0.1$, and
$\lambda_{\text{CVC}}=1e-5$. Check our codebase for more detailed
hyperparameters.

##  Appendix C Discussion

Broader Impact. Our work allows a novice user to create 3D content more
easily. The 3D outputs can be directly used in photo editing software as well
as virtual reality and augmented reality applications.

Similar to recent works on data-driven 2D and 3D face synthesis, we suffer
from biases in the underlying dataset. Our model is trained on CelebAMask-HQ
dataset, as it provides segmentation masks that can be used as conditional
input. To reduce the dataset bias, one future direction is to run our model on
more diverse datasets with a pre-trained face parser. While our work allows
for controllable 3D content generation, there may be potential misuse of the
generated content. As an attempt to identify the generated content from the
real photos, we run a forensics detector [73] on our generated results, and
find our generated images can be detected with an accuracy of $89.77\%$, and
an average precision of $99.97\%$.

Usage of Existing Assets. We use CelebAMask-HQ dataset [38]. The CelebA
dataset is available for non-commercial research purposes only. All images of
the CelebA dataset are obtained from the Internet. The face identities are
released upon request for research purposes only. See CelebA website for
details. We also use AFHQ Cat dataset [16]. This dataset is under Creative
Commons license. Our work is also inspired by a few codebases. StyleNeRF
codebase [21] is under Creative Common license. StyleGAN2 codebase [36] is
under the Nvidia Source Code License. EG3D codebase [8] is under the Nvidia
Source Code License.

Limitations. Our current method has three major limitations. First, it mainly
focuses on modeling the appearance and geometry of a single object category.
Extending the method to more complex scene datasets with multiple objects is a
promising next step, though defining a canonical pose for generic scenes poses
a nontrivial challenge. Second, our model’s generation is limited to the
dataset’s distribution. Our model will not follow the user input unless it is
within the dataset’s distribution. Incorporating diffusion models and training
on more diverse datasets can potentially improve the generalization. Finally,
our model training requires camera poses associated with each training image,
though our method does not require poses during inference time. Eliminating
the requirement for pose information will further broaden the scope of
applications.

##  Appendix D Changelog

V2. Add more citations in Section 2. Fix some typos.

V1. Initial preprint release (CVPR 2023).