# Distributionally robust stochastic optimal control

Alexander Shapiro Georgia Institute of Technology, Atlanta, Georgia 30332,
USA<EMAIL_ADDRESS>
Research of this author was partially supported by Air Force Office of
Scientific Research (AFOSR) under Grant FA9550-22-1-0244. Yan Li Georgia
Institute of Technology, Atlanta, Georgia 30332, USA<EMAIL_ADDRESS>
The main goal of this paper is to discuss the construction of distributionally
robust counterparts of stochastic optimal control problems. Randomized and
non-randomized policies are considered. In particular, necessary and
sufficient conditions for the existence of non-randomized policies are given.

Keywords: stochastic optimal control, dynamic equations distributional
robustness, game formulation, risk measures

##  1 Introduction

Consider the Stochastic Optimal Control (SOC) (discrete time, finite horizon)
model (e.g., [1]):

$\min\limits_{\pi\in\Pi}{\mathbb{E}}^{\pi}\left[\sum_{t=1}^{T}f_{t}(x_{t},u_{t},\xi_{t})+f_{T+1}(x_{T+1})\right],$
(1.1)

where $\Pi$ is the set of polices satisfying the constraints

$\Pi=\Big{\\{}\pi=(\pi_{1},\ldots,\pi_{T}):u_{t}=\pi_{t}(x_{t},\xi_{[t-1]}),u_{t}\in{\cal
U}_{t},x_{t+1}=F_{t}(x_{t},u_{t},\xi_{t}),\;\;t=1,...,T\Big{\\}}.$ (1.2)

Here variables $x_{t}\in{\mathbb{R}}^{n_{t}}$, $t=1,...,T+1$, represent the
state of the system, $u_{t}\in{\mathbb{R}}^{m_{t}}$, $t=1,...,T$, are
controls, $\xi_{t}\in\Xi_{t}$, $t=1,...,T$, are random vectors, $\Xi_{t}$ is a
closed subset of ${\mathbb{R}}^{d_{t}}$,
$f_{t}:{\mathbb{R}}^{n_{t}}\times{\mathbb{R}}^{m_{t}}\times{\mathbb{R}}^{d_{t}}\to{\mathbb{R}}$,
$t=1,...,T$, are cost functions, $c_{T+1}(x_{T+1})$ is a final cost function,
$F_{t}:{\mathbb{R}}^{n_{t}}\times{\mathbb{R}}^{m_{t}}\times{\mathbb{R}}^{d_{t}}\to{\mathbb{R}}^{n_{t+1}}$
are (measurable) mappings and ${\cal U}_{t}$ is a (nonempty) subset of
${\mathbb{R}}^{m_{t}}$. Values $x_{1}$ and $\xi_{0}$ are deterministic
(initial conditions); it is also possible to view $x_{1}$ as random with a
given distribution, this is not essential for the following discussion.

  * •

Unless stated otherwise we assume that the probability law of the random
process $\xi_{1},...,\xi_{T}$ does not depend on our decisions.

The optimization in (1.1) is performed over policies $\pi\in\Pi$ determined by
decisions $u_{t}$ and state variables $x_{t}$ considered as functions of
$\xi_{[t-1]}=(\xi_{1},...,\xi_{t-1})$, $t=1,...,T$, and satisfying the
feasibility constraints (1.2). We also denote
$\Xi_{[t]}:=\Xi_{1}\times\cdots\times\Xi_{t}$. For the sake of simplicity, in
order not to distract from the main message of the paper, we assume that the
control sets ${\cal U}_{t}$ do not depend on $x_{t}$. It is possible to extend
the analysis to the general case, where the control sets are functions of the
state variables.

######  Remark 1.1.

Note that because of the basic assumption that the probability distribution of
$\xi_{1},...,\xi_{T}$ does not depend on our decisions (does not depend on
states and actions), it suffices to consider policies
$\\{\pi_{t}(\xi_{[t-1]})\\}$ as functions of the process $\xi_{t}$ alone. It
also could be noted that in case the random process $\xi_{t}$ is stagewise
independent, i.e., random vector $\xi_{t+1}$ is independent of $\xi_{[t]}$,
$t=1,...,T-1$, it suffices to consider policies of the form
$\\{u_{t}=\pi_{t}(x_{t})\\}$.  $\hfill\square$

Consider Banach space $C(\Xi_{t})$ of continuous functions
$\phi:\Xi_{t}\to{\mathbb{R}}$ equipped with the sup-norm
$\|\phi\|_{\infty}=\sup_{\xi\in\Xi_{t}}|\phi(\xi)|$. The dual space of
$C(\Xi_{t})$ is the space of finite signed measures on $\Xi_{t}$ with respect
to the bilinear form $\langle\phi,\gamma\rangle=\int\phi d\gamma$ (Riesz
representation).

######  Remark 1.2.

Denote by ${\mathfrak{M}}_{t}$ the set of probability measures on the set
$\Xi_{t}$ equipped with its Borel sigma algebra. The set ${\mathfrak{M}}_{t}$
is a weakly∗ closed subset of the unit ball of the dual space of $C(\Xi_{t})$
and hence is weakly∗ compact by the Banach - Alaoglu theorem. The weak∗
topology111In probability theory this weak∗ topology is often referred to as
the weak topology, e.g., [2].  of ${\mathfrak{M}}_{t}$ is metrizable (e.g.,
[6, Theorem 6.30]). It can be noted that if $\phi_{n}\in C(\Xi_{t})$ converges
(in the norm topology) to $\phi\in C(\Xi_{t})$ and
$\gamma_{n}\stackrel{{\scriptstyle\omega^{*}}}{{\to}}\gamma$, then
$\int\phi_{n}d\gamma_{n}=\langle\phi_{n},\gamma_{n}\rangle$ converges to
$\int\phi d\gamma$ (e.g., [3, Proposition 2.24]).  $\hfill\square$

##  2 Distributionally Robust Stochastic Optimal Control

Consider the setting where ambiguity set ${\cal P}_{t}^{\xi_{[t-1]}}$ of
probability measures on $\Xi_{t}$ depends on the history
$\xi_{[t-1]}=(\xi_{1},...,\xi_{t-1})$ (cf., [8]). That is, ${\cal
P}_{t}^{\xi_{[t-1]}}={\mathbb{M}}_{t}(\xi_{[t-1]})$, where
${\mathbb{M}}_{t}:\Xi_{[t-1]}\rightrightarrows{\mathfrak{M}}_{t}$ is the
respective multifunction. Here ${\mathbb{M}}_{1}(\cdot)\equiv{\cal P}_{1}$ for
some pre-determined ${\cal P}_{1}\subset{\mathfrak{M}}_{1}$.

We view the distributionally robust counterpart of problem (1.1) as a dynamic
game between the decision maker (the controller) and the adversary (the
nature). Define history of the decision process as

$\mathfrak{h}_{t}=(x_{1},u_{1},P_{1},\xi_{1},x_{2},...,u_{t-1},P_{t-1},\xi_{t-1},x_{t}),\;t=1,...,T.$
(2.3)

At stage $t=1,...,T$, based on $\mathfrak{h}_{t}$ the nature chooses a
probability measure $P_{t}\in{\cal P}_{t}^{\xi_{[t-1]}}$, at the same time the
controller chooses its action $u_{t}\in{\cal U}_{t}$. For a realization of
$\xi_{t}\sim P_{t}$, the next state $x_{t+1}=F_{t}(x_{t},u_{t},\xi_{t})$ and
so on the process is continued222 It is also possible to let the ambiguity set
${\cal P}_{t}^{\xi_{[t-1]}}$ depend on the past actions
$u_{[t-1]}=(u_{1},\ldots,u_{t-1})$. This does not change the essence of our
discussion for the dynamic game.. We write the corresponding distributionally
robust problem as

$\min\limits_{\pi\in\Pi}\sup_{\gamma\in\Gamma}{\mathbb{E}}^{\pi,\gamma}\left[\sum_{t=1}^{T}f_{t}(x_{t},u_{t},\xi_{t})+f_{T+1}(x_{T+1})\right],$
(2.4)

where $\Pi$ is the set of polices of the controller and $\Gamma$ is the set of
policies of the nature (cf., [10, p. 812]).

Unless stated otherwise we make the following assumptions about ${\cal
U}_{t},\Xi_{t},f_{t},F_{t}$.

######  Assumption 2.1.

The sets ${\cal U}_{t}$ and $\Xi_{t}$, $t=1,...,T$, are compact,
$F_{t}(x_{t},u_{t},\xi_{t})$, $f_{t}(x_{t},u_{t},\xi_{t})$, $t=1,...,T$, and
$f_{T+1}(x_{T+1})$ are continuous.

###  2.1 Dynamic Equations

The dynamic programming equations for the distributionally robust counterpart
(2.4) of problem (1.1) are ${\cal
V}_{T+1}(x_{T+1},\xi_{[T]})=f_{T+1}(x_{T+1})$ for all
$\xi_{[T]}\in\Xi_{1}\times\cdots\times\Xi_{T}$, and for $t=T,...,1$,

$\displaystyle{\cal V}_{t}(x_{t},\xi_{[t-1]})$
$\displaystyle=\inf\limits_{u_{t}\in{\cal U}_{t}}\sup_{P_{t}\in{\cal
P}^{\xi_{[t-1]}}_{t}}{\mathbb{E}}_{\xi_{t}\sim
P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+{\cal
V}_{t+1}\big{(}F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]}\big{)}\right].$ (2.5)

To ensure that (2.5) is well-defined, we need value functions ${\cal V}_{t}$
to be measurable. We establish below the continuity of the value functions and
that the optimal action in (2.5) can be attained. We refer the readers to
Appendix for the the definition of continuous multifunctions.

######  Proposition 2.1.

Suppose that Assumption 2.1 holds and the multifunctions ${\mathbb{M}}_{t}$
are continuous. Then the value functions ${\cal V}_{t}(x_{t},\xi_{[t-1]})$ in
(2.5) are continuous in
$(x_{t},\xi_{[t-1]})\in{\mathbb{R}}^{n_{t}}\times\Xi_{[t-1]}$ and the infimum
in (2.5) is attained. The optimal policy of the controller is given by
$\bar{\pi}_{t}(x_{t},\xi_{[t-1]})=\bar{u}_{t}(x_{t},\xi_{[t-1]})$,
$t=1,...,T,$ with

$\bar{u}_{t}(x_{t},\xi_{[t-1]})\in\mathop{\rm arg\,min}\limits_{u_{t}\in{\cal
U}_{t}}\sup_{P_{t}\in{\cal
P}_{t}^{\xi_{[t-1]}}}{\mathbb{E}}_{P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+{\cal
V}_{t+1}\big{(}F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]}\big{)}\right].$ (2.6)

###### Proof.

We argue by induction in $t=T,...,1$. We have that ${\cal
V}_{T+1}(x_{T+1},\xi_{[T]})=f_{T+1}(x_{T+1})$ and hence ${\cal
V}_{T+1}(x_{T+1},\xi_{[T]})$ is continuous by the assumption. Now suppose that
${\cal V}_{t+1}(\cdot,\cdot)$ is continuous. Consider

$\displaystyle\psi(P_{t},x_{t},u_{t},\xi_{[t-1]})$ $\displaystyle:=$
$\displaystyle{\mathbb{E}}_{\xi_{t}\sim
P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+{\cal
V}_{t+1}(F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]})\right]$ $\displaystyle=$
$\displaystyle\int_{\Xi_{t}}f_{t}(x_{t},u_{t},\xi_{t})+{\cal
V}_{t+1}(F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t-1]},\xi_{t})P_{t}(d\xi_{t}).$

From Assumption 2.1 the set $\Xi_{t}$ is compact, and subsequently for any
sequence $(x_{t}^{n},u_{t}^{n},\xi_{[t-1]}^{n})$ converging to
$(x_{t},u_{t},\xi_{[t-1]})$, we have that
$\phi_{t}^{n}(\xi_{t}):=f_{t}(x^{n}_{t},u_{t}^{n},\xi_{t})+{\cal
V}_{t+1}(F_{t}(x_{t}^{n},u_{t}^{n},\xi_{t}),(\xi_{[t-1]}^{n},\xi_{t}))$
converges to $\phi_{t}(\xi_{t}):=f_{t}(x_{t},u_{t},\xi_{t})+{\cal
V}_{t+1}(F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]})$ uniformly in
$\xi_{t}\in\Xi_{t}$. For any $P_{t}^{n}\overset{w^{*}}{\to}P_{t}$, it follows
by Remark 1.2 that
$\psi_{t}(P_{t}^{n},x_{t}^{n},u_{t}^{n},\xi_{[t-1]}^{n})\to\psi_{t}(P_{t},x_{t},u_{t},\xi_{[t-1]})$.
In addition, since $\Xi_{t}$ is compact, $C(\Xi_{t})$ is separable, and
${\mathfrak{M}}_{t}$ with weak∗ topology is metrizable [6, Theorem 6.30]. It
follows that $\psi_{t}$ is continuous jointly in $P_{t}$ (with respect to the
weak∗ topology of ${\mathfrak{M}}_{t}$) and
$(x_{t},u_{t},\xi_{[t-1]})\in{\mathbb{R}}^{n_{t}}\times{\mathbb{R}}^{m_{t}}\times\Xi_{[t-1]}$.
Consequently we obtain by Theorem A.1 that the function

$\nu(x_{t},u_{t},\xi_{[t-1]}):=\sup_{P_{t}\in{\cal
P}_{t}^{\xi_{[t-1]}}}\psi(P_{t},x_{t},u_{t},\xi_{[t-1]})$

is continuous in $(x_{t},u_{t},\xi_{[t-1]})$. Finally again by Theorem A.1 we
have that

${\cal V}_{t}(x_{t},\xi_{[t-1]})=\inf_{u_{t}\in{\cal
U}_{t}}\nu(x_{t},u_{t},\xi_{[t-1]})$

is continuous. The infimum is attained as ${\cal U}_{t}$ is compact. ∎

######  Remark 2.1.

It can be noted that because of the basic assumption that the probability laws
of the process $\xi_{t}$ do not depend on the states and actions, it follows
that the state $x_{t}$ can be considered as a function of $\xi_{[t-1]}$, and
hence the value functions ${\cal V}_{t}(\xi_{[t-1]})$ and optimal policies
$\bar{\pi}_{t}(\xi_{[t-1]})$ can be viewed as functions of the process
$\xi_{1},...,\xi_{T}$ alone (compare with Remark 1.1).  $\hfill\square$

######  Remark 2.2.

Condition (2.6) is sufficient for the corresponding policy to be optimal. It
could be interested to note that for certain classes of ambiguity sets such
condition is not necessary for the policy to be optimal for problem (2.4) even
if ambiguity sets ${\cal P}_{t}$ do not depend on $\xi_{[t-1]}$ (cf., [9]).
$\hfill\square$

###  2.2 Randomized Policies

We also consider randomized policies for the controller. That is, at stage
$t=1,...,T$, the nature chooses a probability measure $P_{t}\in{\cal
P}_{t}^{\xi_{[t-1]}}$, at the same time the controller chooses its action
$u_{t}$ at random according to a probability distribution
$\mu_{t}\in{\mathfrak{C}}_{t}$, where ${\mathfrak{C}}_{t}$ denotes the set of
(Borel) probability measures on ${\cal U}_{t}$. Consequently
$x_{t+1}=F_{t}(x_{t},u_{t},\xi_{t})$ and so on. Here the decision process is
defined by histories ${\mathfrak{h}}_{t}$, as defined in (2.3), with actions
$u_{t}$ chosen at random.

This corresponds to the min-max formulation

$\min\limits_{\mu\in\widehat{\Pi}}\sup_{\gamma\in\Gamma}{\mathbb{E}}^{\mu,\gamma}\left[\sum_{t=1}^{T}f_{t}(x_{t},u_{t},\xi_{t})+f_{T+1}(x_{T+1})\right],$
(2.7)

where $\widehat{\Pi}$ is the set of randomized policies of the controller that
maps $\mathfrak{h}_{t}$ into ${\mathfrak{C}}_{t}$ at each stage. For the
randomized policies the counterpart of dynamic equation (2.5) becomes
$V_{T+1}(x_{T+1},\xi_{[T]})=f_{T+1}(x_{T+1})$ for all $\xi_{[T]}\in\Xi_{[T]}$,
and for $t=T,...,1$,

$V_{t}(x_{t},\xi_{[t-1]})=\inf\limits_{\mu_{t}\in{\mathfrak{C}}_{t}}\sup_{P_{t}\in{\cal
P}^{\xi_{[t-1]}}_{t}}{\mathbb{E}}_{(u_{t},\xi_{t})\sim\mu_{t}\times
P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+V_{t+1}\big{(}F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]}\big{)}\right]$
(2.8)

(cf., [7],[14]). The controller has a non-randomized optimal policy if for
every $t$ and $x_{t}$ problem (2.9) below has optimal solution supported on a
single point of ${\cal U}_{t}$.

Note that the set ${\mathfrak{C}}_{t}$ is compact in the weak∗ topology of the
dual of the space $C({\cal U}_{t})$. With similar arguments as in Proposition
2.1, we have the following.

######  Proposition 2.2.

Suppose that Assumption 2.1 holds and the multifunctions ${\mathbb{M}}_{t}$
are continuous. Then the value functions $V_{t}(x_{t},\xi_{[t-1]})$ in (2.8)
are continuous in
$(x_{t},\xi_{[t-1]})\in{\mathbb{R}}^{n_{t}}\times\Xi_{[t-1]}$ and the infimum
in (2.8) is attained. The optimal policy of the controller is given by
$\bar{\pi}_{t}(x_{t},\xi_{[t-1]})=\bar{\mu}_{t}(x_{t},\xi_{[t-1]})$ with

$\bar{\mu}_{t}(x_{t},\xi_{[t-1]})\in\mathop{\rm
arg\,min}\limits_{\mu_{t}\in{\mathfrak{C}}_{t}}\sup_{P_{t}\in{\cal
P}^{\xi_{[t-1]}}_{t}}{\mathbb{E}}_{(u_{t},\xi_{t})\sim\mu_{t}\times
P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+V_{t+1}\big{(}F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]}\big{)}\right].$
(2.9)

###### Proof.

We have that $V_{T+1}(x_{T+1},\xi_{[T]})=f_{T+1}(x_{T+1})$ and hence
$V_{T+1}(x_{T+1},\xi_{[T]})$ is continuous by the assumption. Now suppose that
$V_{t+1}(\cdot,\cdot)$ is continuous. Consider

$\displaystyle\psi(P_{t},x_{t},\mu_{t},\xi_{[t-1]})$ $\displaystyle:=$
$\displaystyle{\mathbb{E}}_{(u_{t},\xi_{t})\sim\mu_{t}\times
P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+V_{t+1}(F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]})\right]$
$\displaystyle=$ $\displaystyle\int_{{\cal
U}_{t}}\int_{\Xi_{t}}f_{t}(x_{t},u_{t},\xi_{t})+V_{t+1}(F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]})\mu_{t}(du_{t})P_{t}(d\xi_{t}).$

From Assumption 2.1, both $\Xi_{t}$ and ${\cal U}_{t}$ are compact, and
subsequently for any $(x_{t}^{n},\xi_{[t-1]}^{n})$ converging to
$(x_{t},\xi_{[t-1]})$, we have that
$\phi_{t}^{n}(u_{t},\xi_{t}):=f_{t}(x^{n}_{t},u_{t},\xi_{t})+V_{t+1}(F_{t}(x_{t}^{n},u_{t},\xi_{t}),(\xi_{[t-1]}^{n},\xi_{t}))$
converges to
$\phi_{t}(u_{t},\xi_{t}):=f_{t}(x_{t},u_{t},\xi_{t})+V_{t+1}(F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]})$
uniformly in $(u_{t},\xi_{t})\in{\cal U}_{t}\times\Xi_{t}$. For
$P_{t}^{n}\overset{w^{*}}{\to}P_{t}$ and
$\mu_{t}^{n}\overset{w^{*}}{\to}\mu_{t}$, from compactness of $\Xi_{t}$ and
${\cal U}_{t}$, it can be readily shown that $P_{t}^{n}\times\mu_{t}^{n}$
converges to $P_{t}\times\mu_{t}$ in the weak∗ topology of the dual of
$C(\Xi_{t}\times{\cal U}_{t})$, and hence
$\psi_{t}(P_{t}^{n},x_{t}^{n},\mu_{t}^{n},\xi_{[t-1]}^{n})\to\psi_{t}(P_{t},x_{t},\mu_{t},\xi_{[t-1]})$
(Remark 1.2). In addition, both ${\mathfrak{M}}_{t}$ and ${\mathfrak{C}}_{t}$
with weak∗ topology are metrizable [6, Theorem 6.30]. It follows that
$\psi_{t}$ is continuous jointly in $(P_{t},\mu_{t})$ (with respect to the
product weak∗ topology of ${\mathfrak{M}}_{t}$ and ${\mathfrak{C}}_{t}$) and
$(x_{t},\xi_{[t-1]})\in{\mathbb{R}}^{n_{t}}\times\Xi_{[t-1]}$. The rest of the
proof then follows from the same lines as in Proposition 2.1. ∎

######  Remark 2.3.

Note that here the state $x_{t}$ also depends on the history of chosen
controls $u_{\tau}\sim\mu_{\tau}$, $\tau=1,...,t-1$. Therefore the optimal
policy $\bar{\pi}_{t}(x_{t},\xi_{[t-1]})$ cannot be written as functions of
$\xi_{[t-1]}$ alone (compare with Remark 2.1).  $\hfill\square$

######  Remark 2.4.

An important particular case of the above framework is when the ambiguity sets
${\cal P}_{t}$ do not depend on $\xi_{[t-1]}$. This can be viewed as the
counterpart of the stagewise independence condition. In that case the value
functions $V_{t}(x_{t})$ and optimal policies $\bar{\pi}(x_{t})$ do not depend
on $\xi_{[t-1]}$ and the respective dynamic equations can be written as

$V_{t}(x_{t})=\inf\limits_{\mu_{t}\in{\mathfrak{C}}_{t}}\sup_{P_{t}\in{\cal
P}_{t}}{\mathbb{E}}_{(u_{t},\xi_{t})\sim\mu_{t}\times
P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+V_{t+1}\big{(}F_{t}(x_{t},u_{t},\xi_{t})\big{)}\right].$
(2.10)

Also in that case the assumption of continuity of the multifunctions
${\mathbb{M}}_{t}$ holds automatically. We will discuss this case further in
Section 2.4.  $\hfill\square$

In Section 4 we will discuss necessary and sufficient conditions for the
existence of non-randomized optimal policies for the controller.

###  2.3 Nested Formulation

For non-randomized policies of the controller, dynamic equations (2.5)
correspond to the following nested formulation of the respective
distributionally robust SOC. For a non-randomized policy
$\\{\pi_{t}=\pi_{t}(x_{t},\xi_{[t-1]})\\}$, consider the total cost

$Z^{\pi}:=\sum_{t=1}^{T}f_{t}(x_{t},\pi_{t},\xi_{t})+f_{T+1}(x_{T+1}).$

Recall that we can consider here policies $\pi_{t}(\xi_{[t-1]})$ as functions
of $\xi_{[t-1]}$ alone, and that the state $x_{t}$ is also a function of
$\xi_{[t-1]}$ (see Remark 2.1). Therefore we can view
$Z^{\pi}=Z^{\pi}(\xi_{[T]})$ as a function of the process
$\xi_{1},...,\xi_{T}$.

Consider linear spaces ${\cal Z}_{t}$ of bounded random variables
$Z_{t}:\Xi_{[t]}\to{\mathbb{R}}$. For $t=T,...,1$, define recursively mappings
${\cal R}_{t}:{\cal Z}_{t}\to{\cal Z}_{t-1}$,

${\cal R}_{t}(Z_{t}):=\sup_{P_{t}\in{\cal
P}_{t}^{\xi_{[t-1]}}}\left\\{{\mathbb{E}}_{\xi_{t}\sim
P_{t}}[Z_{t}(\xi_{1},...,\xi_{t-1},\xi_{t})]=\int_{\Xi_{t}}Z_{t}(\xi_{1},...,\xi_{t-1},\xi_{t})dP_{t}(\xi_{t})\right\\}.$
(2.11)

Note that ${\cal Z}_{0}={\mathbb{R}}$ and ${\cal
R}_{1}(Z_{1})=\sup_{P_{1}\in{\cal P}_{1}}{\mathbb{E}}_{P_{1}}[Z_{1}]$ is a
real number. Let

${\mathfrak{R}}_{T}(Z_{T}):={\cal R}_{1}({\cal R}_{2}(\cdots{\cal
R}_{T}(Z_{T})))$ (2.12)

be the corresponding composite functional. Then with $Z_{T}=Z^{\pi}$ the
nested distributionally robust counterpart of problem (1.1) is

$\begin{array}[]{ll}\min\limits_{\pi\in\Pi}{\mathfrak{R}}_{T}\left[\sum_{t=1}^{T}f_{t}(x_{t},\pi_{t},\xi_{t})+f_{T+1}(x_{T+1})\right],\end{array}$
(2.13)

where (as before) $\Pi$ is the set of non-randomized policies of the
controller.

###  2.4 Stagewise Independence Setting

When ${\mathbb{M}}_{t}(\cdot)\equiv{\cal P}_{t}$ (i.e., ${\cal
P}_{t}^{\xi_{[t-1]}}={\cal P}_{t}$ for all $\xi_{[t-1]}\in\Xi_{[t-1]}$, see
Remark 2.4), we can consider the static counterpart of problem (2.4) defined
as

$\min\limits_{\pi\in\Pi}\sup_{P\in{\mathfrak{P}}}{\mathbb{E}}^{\pi}_{\xi_{[T]}\sim
P}\left[\sum_{t=1}^{T}f_{t}(x_{t},u_{t},\xi_{t})+f_{T+1}(x_{T+1})\right],$
(2.14)

where

${\mathfrak{P}}:=\\{P=P_{1}\times\cdots\times P_{T}:P_{t}\in{\cal
P}_{t},\;t=1,...,T\\}.$ (2.15)

Here the nature chooses the probabilities prior to realizations of the process
$\xi_{1},...,\xi_{T}$. In this case, two formulations (2.4) and (2.14) are
equivalent if for any policy $\pi\in\Pi$ of the controller there are optimal
probabilities $P^{*}_{t}$ in (2.5), with $u_{t}=\pi_{t}(x_{t},\xi_{[t-1]})$,
independent of the states $x_{t}$. This, however, happens only in rather
exceptional cases (cf., [8]). The ambiguity set of probability distributions
of $\xi_{[T]}$ of the form (2.15) can be viewed as the counterpart of the
stagewise-independence condition in the distributionally robust setting.

##  3 Duality of Game Formulation

The dynamic equations of the dual of problem (2.7) are:
$Q_{T+1}(x_{T+1},\xi_{[T]})=f_{T+1}(x_{T+1})$ for all $\xi_{[T]}\in\Xi_{[T]}$,
and for $t=T,...,1$,

$Q_{t}(x_{t},\xi_{[t-1]})=\sup_{P_{t}\in{\cal
P}_{t}^{\xi_{[t-1]}}}\inf\limits_{\mu_{t}\in{\mathfrak{C}}_{t}}{\mathbb{E}}_{\mu_{t}\times
P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+Q_{t+1}\big{(}F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]}\big{)}\right].$
(3.1)

Note that for a given $P_{t}$ it suffices to perform the minimization in (3.1)
over Dirac measures333We denote by $\delta_{a}$ the Dirac measure of mass one
at point $a$. $\mu_{t}=\delta_{u_{t}}$, and hence we can write equation (3.1)
in the following equivalent way

$Q_{t}(x_{t},\xi_{[t-1]})=\sup_{P_{t}\in{\cal
P}_{t}^{\xi_{[t-1]}}}\inf\limits_{u_{t}\in{\cal
U}_{t}}{\mathbb{E}}_{P_{t}}\left[f_{t}(x_{t},u_{t},\xi_{t})+Q_{t+1}\big{(}F_{t}(x_{t},u_{t},\xi_{t}),\xi_{[t]}\big{)}\right].$
(3.2)

Similar to Proposition 2.2, it can be shown that the value functions
$Q_{t}(\cdot)$ are well-defined and continuous. By the standard theory of min-
max, we have that $Q_{t}(\cdot)\leq V_{t}(\cdot)$. We next establish that
equality indeed holds when the multifunctions ${\mathbb{M}}_{t}$, $t=1,...,T$,
are convex-valued, i.e., the sets ${\cal
P}_{t}^{\xi_{[t-1]}}={\mathbb{M}}_{t}(\xi_{[t-1]})$ are convex for all
$\xi_{[t-1]}$.

######  Proposition 3.1.

Suppose that Assumption 2.1 holds, and the multifunctions ${\mathbb{M}}_{t}$
are continuous and convex-valued, then $Q_{t}(\cdot)=V_{t}(\cdot)$ for all
$t=1,\ldots,T$.

###### Proof.

Under the specified assumptions, the sets ${\cal P}_{t}^{\xi_{[t-1]}}$ and
${\mathfrak{C}}_{t}$ are weakly∗ compact and convex, and of course the
expectation ${\mathbb{E}}_{\mu_{t}\times P_{t}}$ is linear with respect to
$\mu_{t}$ and $P_{t}$. Then by using Sion’s duality theorem (cf., [13]) and
applying induction backward in time, we can conclude that
$Q_{t}(\cdot)=V_{t}(\cdot)$, $t=1,...,T$. ∎

It could be worth mentioning here that the duality of the game formulation
(when considering randomized policies $\widehat{\Pi}$ of the controller) does
not require the convexity of ${\cal U}_{t}$ or any assumptions on the
transition functions $F_{t}$.

##  4 Existence of Non-randomized Optimal Policies

We have the following necessary and sufficient condition for the existence of
a non-randomized optimal policy of the controller (cf., [7, Theorem 2.2]).

######  Theorem 4.1.

Suppose that Assumption 2.1 holds, the multifunctions ${\mathbb{M}}_{t}$ are
continuous and convex-valued. Then the controller has a non-randomized optimal
policy iff the min-max problem in the right hand side of (2.5) has a saddle
point for all $t=1,...,T$ and $(x_{t},\xi_{[t-1]})$.

###### Proof.

Note that the min-max problem (2.5) (resp. the max-min problem (3.2)) has a
saddle point iff $Q_{t}(\cdot)={\cal V}_{t}(\cdot)$, $t=1,...,T$.

It is clear that $V_{t}(\cdot)\leq{\cal V}_{t}(\cdot)$, and $Q_{t}(\cdot)\leq
V_{t}(\cdot)$. Suppose that for $t=1,...,T$ and all $(x_{t},\xi_{[t-1]})$, the
min-max problem (2.5) possesses a saddle point $(\bar{u}_{t},P_{t})$ possibly
depending on $(x_{t},\xi_{[t-1]})$. Then we have ${\cal
V}_{t}(\cdot)=Q_{t}(\cdot)$ and subsequently $V_{t}(\cdot)={\cal
V}_{t}(\cdot)$. It follows that $\bar{\mu}_{t}=\delta_{\bar{u}_{t}}$ is an
optimal solution of the min-max problem (2.8) and hence a non-randomized
optimal policy of the controller.

Conversely, suppose that the controller has a non-randomized optimal policy.
Then $V_{t}(\cdot)={\cal V}_{t}(\cdot)$, $t=1,...,T$. From Proposition 3.1, we
also have that $V_{t}(\cdot)=Q_{t}(\cdot)$, and hence $Q_{t}(\cdot)={\cal
V}_{t}(\cdot)$, $t=1,...,T$. This implies the existence of a saddle point for
(2.5). ∎

In particular, we have the following result (cf., [5]).

######  Corollary 4.1.

Suppose that the sets ${\cal U}_{t}$ are convex, for every $\xi_{t}\in\Xi_{t}$
the functions $f_{t}(x_{t},u_{t},\xi_{t})$ are convex in $(x_{t},u_{t})$, the
mappings
$F_{t}(x_{t},u_{t},\xi_{t})=A_{t}(\xi_{t})x_{t}+B(\xi_{t})u_{t}+b_{t}(\xi_{t})$
are affine, the multifunction ${\mathbb{M}}_{t}$ are continuous and convex-
valued, $t=1,...,T$, and Assumption 2.1 holds. Then the value functions
$V_{t}(x_{t},\xi_{[t-1]})={\cal V}_{t}(x_{t},\xi_{[t-1]})$ are convex in
$x_{t}$ for every $\xi_{[t-1]}$, and the controller has a non-randomized
optimal policy.

###### Proof.

It can be directly verified that under the specified assumptions, the min-max
problem (2.5) has a saddle point, and the value functions ${\cal
V}_{t}(\cdot,\xi_{[t-1]})$ are convex for all $t=1,\ldots T$ and
$\xi_{[t-1]}\in\Xi_{[t-1]}$. ∎

#### Inventory Model.

As an example, consider the following inventory model (cf., [15])

$\begin{array}[]{cll}\min&{\mathbb{E}}\left[\sum_{t=1}^{T}c_{t}u_{t}+\psi_{t}(x_{t},u_{t},D_{t})\right]\\\
{\rm s.t.}&u_{t}\in{\cal
U}_{t},\;\;x_{t+1}=x_{t}+u_{t}-D_{t},\;t=1,...,T,\end{array}$ (4.1)

where
$\psi_{t}(x_{t},u_{t},d_{t}):=b_{t}[d_{t}-(x_{t}+u_{t})]_{+}+h_{t}[x_{t}+u_{t}-d_{t}]_{+},$
$D_{1},...,D_{T}$ is a (random) demand process, $c_{t},b_{t},h_{t}$ are the
ordering, backorder penalty and holding costs per unit, respectively, $x_{t}$
is the inventory level, $u_{t}$ is the order quantity at time $t$, and ${\cal
U}_{t}=[0,a_{t}]$. Suppose that the distribution of $(D_{1},...,D_{T})$ is
supported on $\Xi_{1}\times\cdots\times\Xi_{T}$ with $\Xi_{t}$, $t=1,...,T$,
being a finite subinterval of the nonnegative real line.

The distributionally robust counterpart of (4.1) is a convex problem provided
the respective multifunctions ${\mathbb{M}}_{t}$ are continuous and convex
valued. It follows by Corollary 4.1 that there is a non-randomized optimal
policy of the controller.

##  5 Construction of Ambiguity Sets

There are several ways how the ambiguity sets can be constructed. We discuss
now an approach which can be considered as an extension of the risk averse
modeling of multistage stochastic programs.

Let ${\mathbb{P}}$ be a probability measure on a (closed) set
$\Xi\subset{\mathbb{R}}^{d}$, and ${\cal P}$ be a set of probability measures
on $\Xi$ absolutely continuous with respect to ${\mathbb{P}}$. Consider the
corresponding distributionally robust functional

${\cal R}(Z):=\sup_{P\in{\cal P}}{\mathbb{E}}_{P}[Z]$ (5.1)

defined on an appropriate space ${\cal Z}$ of random variables
$Z:\Xi\to{\mathbb{R}}$. Such functional can be considered as a coherent risk
measure (e.g., [11, Chapter 6]). The functional ${\cal R}$ is law invariant if
${\cal R}(Z)={\cal R}(Z^{\prime})$ when $Z$ and $Z^{\prime}$ are
distributionally equivalent, i.e., ${\mathbb{P}}(Z\leq
z)={\mathbb{P}}(Z^{\prime}\leq z)$ for any $z\in{\mathbb{R}}$. The law
invariant coherent risk measure can be considered as a function of the
cumulative distribution function (cdf) $H_{Z}(z):={\mathbb{P}}(Z\leq z)$.

An important example of law invariant coherent risk measure is the Average
Value-at-Risk

${\sf AV@R}_{\alpha}(Z):=(1-\alpha)^{-1}\int_{\alpha}^{1}{\sf
V@R}_{\tau}(Z)d\tau,\;\alpha\in[0,1),$

where ${\sf V@R}_{\tau}(Z)=\inf\\{z:{\mathbb{P}}_{t}(Z\leq z)\geq\tau\\}$. It
has dual representation of the form (5.1) with444The notation $P\ll Q$ means
that measure $P$ is absolutely continuous with respect to measure $Q$.

${\cal
P}=\left\\{P\ll{\mathbb{P}}:dP/d{\mathbb{P}}\leq(1-\alpha)^{-1}\right\\}.$

Now let ${\mathbb{P}}$ be a reference probability measure on
$\Xi_{[T]}=\Xi_{1}\times\cdots\times\Xi_{T}$. Denote by ${\mathbb{P}}_{t}$ and
${\mathbb{P}}_{[t]}$ the corresponding marginal probability measures on
$\Xi_{t}$ and $\Xi_{[t]}$. Let us define ${\mathbb{P}}_{t}^{\xi_{[t-1]}}$,
with respect to $\Xi_{[t]}=\Xi_{[t-1]}\times\Xi_{t}$ and
${\mathbb{P}}_{[t-1]}$, recursively going backward in time. That is, using the
Regular Probability Kernel (RPK) (e.g., [4, III- 70]), define
${\mathbb{P}}_{t}^{\xi_{[t-1]}}$ as a probability measure on $\Xi_{t}$ for
almost every (a.e.) $\xi_{[t-1]}\in\Xi_{[t-1]}$ (a.e. with respect to
${\mathbb{P}}_{[t-1]}$), and for any measurable sets $A\subset\Xi_{[t-1]}$ and
$B\subset\Xi_{t}$,

${\mathbb{P}}(A\times
B)=\int_{A}{\mathbb{P}}_{t}^{\xi_{[t-1]}}(B)d{\mathbb{P}}_{[t-1]}(\xi_{[t-1]}).$
(5.2)

The conditional counterpart ${\cal R}_{t}^{\xi_{[t-1]}}$ of the law invariant
coherent risk measure can be defined as the respective function of the
conditional cdf of $Z$ (cf., [12]). For example, the conditional counterpart
of the Average Value-at-Risk can be defined in that way and is given by

${\sf AV@R}^{\xi_{[t-1]}}_{\alpha}(Z)=(1-\alpha)^{-1}\int_{\alpha}^{1}{\sf
V@R}_{\tau|\xi_{[t-1]}}(Z)d\tau,\;\alpha\in[0,1),$

where ${\sf V@R}_{\tau|\xi_{[t-1]}}$ is the conditional counterpart of ${\sf
V@R}_{\tau}$.

In turn this defines the set ${\cal P}_{t}^{\xi_{[t-1]}}$ of conditional
probability measures. The respective one-step mappings ${\cal R}_{t}:{\cal
Z}_{t}\to{\cal Z}_{t-1}$, defined in (2.11), are given by

$[{\cal R}_{t}(Z_{t})](\xi_{[t-1]})={\cal R}_{t}^{\xi_{[t-1]}}(Z_{t}).$ (5.3)

This leads to the respective nested functional ${\mathfrak{R}}_{T}$, defined
in (2.12), and the nested formulation (2.13) of the problem with respect to
non-randomized policies of the controller.

In this construction of nested counterparts of law invariant risk measures it
is essential that the ambiguity sets consist of probability measures
absolutely continuous with respect to the reference measure. However, the
above approach can be extended beyond that setting. That is, the conditional
set ${\cal P}_{t}^{\xi_{[t-1]}}$ of probability measures on $\Xi_{t}$ can be
defined in some away with respect to the conditional distribution
${\mathbb{P}}_{t}^{\xi_{[t-1]}}$. For example, ${\cal P}_{t}^{\xi_{[t-1]}}$
can consist of probability measures $P_{t}\in{\mathfrak{M}}_{t}$ with a
Wasserstein distance from ${\mathbb{P}}_{t}^{\xi_{[t-1]}}$ less than or equal
to a constant $r_{t}>0$.

Of course, in such construction it should be verified that the dynamic
programming equations (2.5) (in the case of non-randomized policies) and (2.8)
(in the case of randomized policies) are well defined. There are two important
cases where the corresponding multifunctions ${\mathbb{M}}_{t}$ are continuous
and hence the value functions are continuous (Proposition 2.2). One such case
is when the ambiguity sets ${\cal P}_{t}$ do not depend on $\xi_{[t-1]}$ (see
Section 2.4). This corresponds to the case where ${\mathbb{P}}$ is given by
the direct product ${\mathbb{P}}_{1}\times\cdots\times{\mathbb{P}}_{t}$ of the
marginal measures. In that case, the ambiguity sets ${\cal
P}_{t}\subset{\mathfrak{M}}_{t}$ can be arbitrary. For example, ${\cal P}_{t}$
can be the set of probability measures with Wasserstein distance from the
reference measure ${\mathbb{P}}_{t}$, less than or equal to a positive
constant $r_{t}$. The dynamic programming equations take the form (2.10), and
by Proposition 2.2 the value functions $V_{t}(x_{t})$ are continuous.

Another important case is when the sets $\Xi_{t}$, $t=1,...,T$, are finite,
i.e., $(\xi_{1},...,\xi_{T})$ has discrete distribution with a finite support.
Then continuity of the multifunctions ${\mathbb{M}}_{t}$ trivially holds, and
hence the value functions $V_{t}(x_{t},\xi_{[t-1]})$ are continuous.

##  6 Conclusions

The main technical assumption used in the considered construction is
continuity of the multifunctions ${\mathbb{M}}_{t}$. This assumption ensures
continuity of the value functions and hence measurability of the objective
functions in the dynamic programming equations (2.5) and (2.8). It holds
automatically in the two important cases, namely when the ambiguity sets do
not depend on the history of the process $\xi_{t}$ or when the sets $\Xi_{t}$
are finite. In general it could be difficult to verify that assumption and the
measurability question remains open.

By using the game formulation it is possible to consider the randomized
policies of the controller. In Theorem 4.1 we give necessary and sufficient
conditions for existence of non-randomized optimal policies. The assumption
that the multifunctions ${\mathbb{M}}_{t}$ are convex-valued, i.e., that the
respective ambiguity sets are convex, is rather mild. The assumption of
continuity of ${\mathbb{M}}_{t}$ is needed in order to ensure that the
respective dynamic equations are well defined.

## References

  * [1] D.P. Bertsekas and S.E. Shreve.  Stochastic Optimal Control, The Discrete Time Case.  Academic Press, New York, 1978.
  * [2] P. Billingsley.  Probability and Measure, Third Edition.  Wiley, 1995.
  * [3] J. Frédéric Bonnans and Alexander Shapiro.  Perturbation Analysis of Optimization Problems.  Springer Series in Operations Research. Springer, 2000.
  * [4] Dellacherie C. and Meyer P.A.  Probabilities and Potential.  Holland Publishing, Amsterdam, Netherlands, 1988.
  * [5] E. Delage, D. Kuhn, and W. Wiesemann.  ”Dice”-sion - making under uncertainty: When can a random decision reduce risk?  Management Science, 65(7):3282 – 3301, 2019.
  * [6] A Hitchhiker’s Guide.  Infinite dimensional analysis.  Springer, 2006.
  * [7] Yan Li and A. Shapiro.  Rectangularity and duality of distributionally robust markov decision processes.  https://arxiv.org/abs/2308.11139, 2023.
  * [8] A. Shapiro.  Rectangular sets of probability measures.  Operations Research, 64(2):528–541, 2016.
  * [9] A. Shapiro.  Interchangeability principle and dynamic equations in risk averse stochastic programming.  Operations Research Letters, 45:377–381, 2017.
  * [10] A. Shapiro.  Distributionally robust optimal control and MDP modeling.  Operations Research Letters, 49:809–814, 2021.
  * [11] A. Shapiro, D. Dentcheva, and A. Ruszczyński.  Lectures on Stochastic Programming: Modeling and Theory.  SIAM, Philadelphia, third edition, 2021.
  * [12] A. Shapiro and A. Pichler.  Conditional distributionally robust functionals.  Operations Research, online:1–13, 2023.
  * [13] M. Sion.  On general minimax theorems.  Pacific Journal of Mathematics, 8:171–176, 1958.
  * [14] S. Wang, N. Si, J. Blanchet, and Z. Zhou.  On the foundation of distributionally robust reinforcement learning.  https://arxiv.org/abs/2311.09018, 2023.
  * [15] P.H. Zipkin.  Foundation of inventory management.  McGraw-Hill, Boston, MA, 2000.

##  Appendix A Appendix

Let ${\cal X}$ and $\Gamma$ be nonempty compact metric spaces, $f:{\cal
X}\times\Gamma\to{\mathbb{R}}$ be continuous real valued unction, and
$\Phi:\Gamma\rightrightarrows{\cal X}$ be a multifunction (point-to-set
mapping). It is said that the multifunction $\Phi$ is closed if
$\gamma_{n}\to\gamma$, $x_{n}\in\Phi(\gamma_{n})$ and $x_{n}\to x$ implies
that $x\in\Phi(\gamma)$. If $\Phi$ is closed, then $\Phi$ is closed valued,
i.e., the set $\Phi(\gamma)$ is closed for every $\gamma\in\Gamma$. It is said
that the multifunction $\Phi$ is upper semicontinuous if for any
$\bar{\gamma}\in\Gamma$ and any neighborhood ${\cal V}$ of
$\Phi(\bar{\gamma})$ there is a neighborhood ${\cal U}$ of $\gamma$ such that
$\Phi(\gamma)\subset{\cal V}$ for any $\gamma\in{\cal U}$. In the considered
framework of compact sets, the multifunction $\Phi$ is closed iff it is closed
valued and upper semicontinuous (e.g., [3, Lemma 4.3]).

It is said that the multifunction $\Phi$ is lower semicontinuous if for any
$\bar{\gamma}\in\Gamma$ and any neighborhood ${\cal V}$ of
$\Phi(\bar{\gamma})$ there is a neighborhood ${\cal U}$ of $\gamma$ such that
${\cal V}\cap\Phi(\gamma)\neq\emptyset$ for any $\gamma\in{\cal U}$. It is
said that $\Phi$ is continuous if it is closed and lower semicontinuous.

Consider value function

$v(\gamma):=\sup_{x\in\Phi(\gamma)}f(x,\gamma).$ (A.1)

We have the following result (e.g., [3, Proposition 4.4]).

######  Theorem A.1.

Suppose that the sets ${\cal X}$ and $\Gamma$ are compact, the function
$f:{\cal X}\times\Gamma\to{\mathbb{R}}$ is continuous, and the multifunction
$\Phi:\Gamma\rightrightarrows{\cal X}$ is continuous. Then the value function
$v(\gamma)$ is real valued and continuous.


firstrgb0.1, 0.42, 0.1 secondrgb0.0, 0.35, 0.56

# DiffRF: Rendering-Guided 3D Radiance Field Diffusion

Norman Müller1,2 Yawar Siddiqui1,2 Lorenzo Porzi2 Lorenzo Porzi2 Samuel Rota
Bulò2
Peter Kontschieder2 Matthias Nießner1
Technical University of Munich1 Meta Reality Labs Zurich2

###### Abstract

We introduce DiffRF, a novel approach for 3D radiance field synthesis based on
denoising diffusion probabilistic models. While existing diffusion-based
methods operate on images, latent codes, or point cloud data, we are the first
to directly generate volumetric radiance fields. To this end, we propose a 3D
denoising model which directly operates on an explicit voxel grid
representation. However, as radiance fields generated from a set of posed
images can be ambiguous and contain artifacts, obtaining ground truth radiance
field samples is non-trivial. We address this challenge by pairing the
denoising formulation with a rendering loss, enabling our model to learn a
deviated prior that favours good image quality instead of trying to replicate
fitting errors like floating artifacts. In contrast to 2D-diffusion models,
our model learns multi-view consistent priors, enabling free-view synthesis
and accurate shape generation. Compared to 3D GANs, our diffusion-based
approach naturally enables conditional generation such as masked completion or
single-view 3D synthesis at inference time. ††Project page:
https://sirwyver.github.io/DiffRF/.

Figure 1:  Our method performs denoising of a probabilistic diffusion process
applied to 3D radiance fields. Guided by 3D supervision and volumetric
rendering, our model enables unconditional synthesis of high-fidelity 3D
assets (left). We further introduce the novel application of masked completion
(right), _i.e_., the task of recovering shape and appearance from incomplete
objects (highlighted in light-blue on the top right chair), solved by our
model as conditional inference without task-specific training.

##  1 Introduction

In recent years, Neural Radiance Fields (NeRFs) [mildenhall2020nerf] have
emerged as a powerful representation for fitting individual 3D scenes from
posed 2D input images. The ability to photo-realistically synthesize novel
views from arbitrary viewpoints while respecting the underlying 3D scene
geometry has the potential to disrupt and transform applications like AR/VR,
gaming, mapping, navigation, _etc_. A number of recent works have introduced
extensions for making NeRFs more sophisticated, by _e.g_., showing how to
incorporate scene semantics [kundu2022panoptic, fu2022panoptic], training
models from heterogeneous data sources [martinbrualla2020nerfw], or scaling
them up to represent large-scale scenes [tancik2022blocknerf,
Turki_2022_CVPR]. These advances are testament to the versatility of ML-based
scene representations; however, they still fit to specific, individual scenes
rather than generalizing beyond their input training data.

In contrast, neural field representations that generalize to multiple object
categories or learn priors for scenes across datasets appear much more limited
to date, despite enabling applications like single-image 3D object generation
[chanmonteiro2020pi-GAN, schwarz2020graf, yu2021pixelnerf, Muller_2022_CVPR,
Wang_2021_CVPR] and unconstrained scene exploration [DeVries_2021_ICCV]. These
methods explore ways to disentangle object priors into shape and appearance-
based components, or to decompose radiance fields into several small and
locally-conditioned radiance fields to improve scene generation quality;
however, their results still leave significant gaps w.r.t. photorealism and
geometric accuracy.

Directions involving generative adversarial networks (GANs) that have been
extended from the 2D domain to 3D-aware neural fields generation are
demonstrating impressive synthesis results [Chan2022]. Like regular 2D GANs,
the training objective is based on discriminating 2D images, which are
obtained by rendering synthesized 3D radiance fields.

At the same time, diffusion-based models [sohl2015deep] have recently taken
the computer vision research community by storm, performing on-par or even
surpassing GANs on multiple 2D benchmarks, and are producing photo-realistic
images that are almost indistinguishable from real photographs. For multi-
modal or conditional settings such as text-to-image synthesis, we currently
observe unprecedented output quality and diversity from diffusion-based
approaches. While several works address purely geometric representations
[zhou20213d, luo2021diffusion], lifting the denoising-diffusion formulation
directly to 3D volumetric radiance fields remains challenging. The main reason
lies in the nature of diffusion models, which require a one-to-one mapping
between the noise vector and the corresponding ground truth data samples. In
the context of radiance fields, such volumetric ground truth data is
practically infeasible to obtain, since even running a costly per-sample NeRF
optimization results in incomplete and imperfect radiance field
reconstructions.

In this work, we present the first diffusion-based generative model that
directly synthesizes 3D radiance fields, thus unlocking high-quality 3D asset
generation for both shape and appearance. Our goal is to learn such a
generative model trained across objects, where each sample is given by a set
of posed RGB images.

To this end, we propose a 3D denoising model directly operating on an explicit
voxel grid representation (Fig. 1, left) producing high-frequency noise
estimates. To address the ambiguous and imperfect radiance field
representation for each training sample, we propose to bias the noise
prediction formulation from Denoising Diffusion Probabilistic Models (DDPMs)
towards synthesizing higher image quality by an additional volumetric
rendering loss on the estimates. This enables our method to learn radiance
field priors less prone to fitting artifacts or noise accumulation during the
sampling process. We show that our formulation leads to diverse and
geometrically-accurate radiance field synthesis producing efficient,
realistic, and view-consistent renderings. Our learned diffusion prior can be
applied in an unconditional setting where 3D object synthesis is obtained in a
multi-view consistent way, generating highly-accurate 3D shapes and allowing
for free-view synthesis. We further introduce the new task of conditional
masked completion – analog to shape completion – for radiance field completion
at inference time. In this setting, we allow for realistic 3D completion of
partially-masked objects without the need for task-specific model adaptation
or training (see Fig. 1, right).

We summarize our contributions as follows:

  * •

To the best of our knowledge, we introduce the first diffusion model to
operate directly on 3D radiance fields, enabling high-quality, truthful 3D
geometry and image synthesis.

  * •

We introduce the novel application of 3D radiance field masked completion,
which can be interpreted as a natural extension of image inpainting to the
volumetric domain.

  * •

We show compelling results in unconditional and conditional settings, _e.g_.,
by improving over GAN-based approaches on image quality (from 16.54 to 15.95
in FID) and geometry synthesis (improving MMD from 5.62 to 4.42), on the
challenging PhotoShape Chairs dataset [photoshape2018].

##  2 Related work

Diffusion models. Since the seminal work by Sohl-Dickstein  _et al_.
[sohl2015deep] on generative diffusion modeling, two classes of generative
models have been proposed that perform inversion of a diffusion process:
Denoising Score Matching (DSM) [song2020denoising, song2020improved, genmodel]
and Denoising Diffusion Probablistic Models (DDPMs) [ho2020denoising]. Both
approaches have shown to be flavours of a single framework, coined Score SDE,
in the work of Song  _et al_. [song2020score]. The term “Diffusion models” is
now being used as an all-encompassing name for this constellation of methods.
Several main directions being studied include devising different sampling
schemes [Karras2022edm, song2021denoising] and noising models [daras2022soft,
hoogeboom2022blurring], exploring alternative formulations and training
algorithms [song2021maximum, huang2021variational, nichol2021improved], and
improving efficiency [rombach2021highresolution]. An overview of current
research results is given by Karras  _et al_. [Karras2022edm].

Diffusion models have been used to obtain state-of-the-art results in many
domains such as text-to-image and guided synthesis [GLIDE,
rombach2021highresolution, meng2022sdedit, poole2022dreamfusion], 3D shape
generation [zhou20213d, luo2021diffusion, gaishape, zeng2022lion], molecule
prediction [trippe2022diffusion, xu2022geodiff, luo2021predicting], and video
generation [ho2022video, yang2022diffusion]. Interestingly, diffusion models
have shown to outperform Generative Adversarial Networks (GANs) in high-
resolution image generation tasks [dhariwal2021diffusion,
rombach2021highresolution], achieving unprecedented results in conditional
image generation [ramesh2022hierarchical]. Furthermore, compared to GANs,
which are often prone to divergence and mode collapse [brock2018large,
miyato2018spectral], diffusion models have been observed to be much easier to
train, although training time is still relatively long.

3D generation. Initially developed for 2D image synthesis, adversarial
approaches have also found success in 3D, _e.g_., for generating meshes
[wu2016learning, gao2022get3d], 3D textures [siddiqui2022texturify], voxelized
representations [chen2018text2shape, henzler2019escaping], or Neural Radiance
Fields (NeRFs) [chanmonteiro2020pi-GAN, schwarz2020graf, niemeyer2021giraffe,
Chan2022, gu2021stylenerf, zhou2021cips]. In particular, methods in this last
category have received much attention in recent years, as they can be trained
purely from collections of 2D images, without any form of 3D supervision, and
enable for the first time photo-realistic novel view synthesis of the
generated 3D objects. Pi-GAN [chanmonteiro2020pi-GAN] and GRAF
[schwarz2020graf] propose similar approaches where a standard NeRF
[mildenhall2020nerf] model is cast in a GAN setting by adding a form of
stochastic conditioning, trained with an adversarial loss. These approaches
are partly limited by the high training-time memory cost of NeRF-style
volumetric rendering, forcing them to use low-resolution image patches.
CIPS-3D [zhou2021cips] and GIRAFFE [niemeyer2021giraffe] solve this issue by
letting the volume rendering component output a low-resolution 2D feature map,
which is then upsampled by an efficient convolutional network to produce the
final image. This approach drastically improves the quality and resolution of
the rendered images, but also introduces 3D inconsistencies, as the
convolutional stage can process different views of the same object in
arbitrarily different ways. StyleNeRF [gu2021stylenerf] partially addresses
this problem by carefully designing the convolutional stage to minimize
inconsistencies, while EG3D [Chan2022] further improves on training efficiency
by replacing the MLP-based NeRF with a light-weight tri-plane volumetric model
generated by a convolutional network. In contrast to our method, these GAN-
based approaches do not naturally support conditional synthesis or completion.

Compared to GANs, diffusion models are relatively under-explored as a tool for
3D synthesis, but a few works have emerged in the past two years. Some
diffusion based-generators have been proposed for 3D point clouds [zhou20213d,
zeng2022lion, luo2021diffusion, gaishape], showing promising results for
conditional synthesis, completion, and other related tasks [zeng2022lion,
zhou20213d]. DreamFusion [poole2022dreamfusion], 3DDesigner
[li20223ddesigner], and GAUDI [bautista2022gaudi], like our work, employ
diffusion models in conjunction with radiance fields, with applications in
both conditional (on text and images) and unconditional 3D generation.
DreamFusion [poole2022dreamfusion] presents an algorithm to generate NeRFs,
augmented with an illumination component, by optimizing a loss defined by a
pre-trained 2D text-conditional diffusion model. GAUDI [bautista2022gaudi]
builds a 3D scene generator by first training a conditional NeRF to
reconstruct a set of indoor videos given scene-specific latents, and then
fitting a diffusion model to capture the learned latent space. Concurrent
works like [anciukevicius2022renderdiffusion, wang2022rodin] apply the
denoising-diffusion approach to factorized radiance representations. In
contrast, our diffusion model operates directly in the space of radiance
fields which directly enables 3D-conditional tasks like shape completion (Sec.
4.2) by leveraging the learned volumetric prior.

##  3 Method

Our method consists of a generative model for 3D objects that builds on recent
state-of-the-art diffusion probabilistic models [ho2020denoising]. It is
trained to revert a process that gradually corrupts 3D objects by injecting
noise at different scales. In our case, 3D objects are represented as radiance
fields [mildenhall2020nerf], so the learned denoising process allows our
method to generate object radiance fields from noise.

Figure 2:  For a time step $t$ uniformly sampled from ${1,...,T}$, we first
diffuse an initial radiance field $f_{0}$ according to a fixed noising
schedule. The resulting $f_{t}$ is passed through a time-conditioned 3D-UNet,
giving an estimate of the applied noise $\epsilon$. We guide the model by the
noise prediction loss $L_{\mathtt{RF}}$ as well as a rendering loss
$L_{\mathtt{RGB}}$ on the predicted denoising $\tilde{f}_{0}$.

Since we target the generation of 3D objects as radiance fields, we begin with
a brief overview of this representation before delving into the details of our
method.

###  3.1 Radiance Fields

A radiance field $(\sigma,\xi)$ is an implicit, volumetric representation of a
3D object that is given in terms of a density field
$\sigma:\mathcal{X}\to\mathbb{R}_{+}$ and an RGB color field
$\xi:\mathcal{X}\to\mathbb{R}^{3}$ defined over a 3D domain
$\mathcal{X}\subset\mathbb{R}^{3}$.111Typically the color field spans also the
viewing direction, but we omit it in this work. The density field gives
information about the presence of an object in a specific point in space,
whereas the color field provides the eventual corresponding RGB color.

Following a ray casting logic, the radiance field can be rendered along a
given ray $r$ yielding an RGB color $c_{r}$ with the following equation
[mildenhall2020nerf]

$c_{r}\coloneqq\int_{0}^{\infty}\tau_{s}(r)\sigma(r_{s})\xi(r_{s})ds\,,$ (1)

where a ray $r$ is a linear curve parametrized by $s$ with unit velocity,
$r_{s}\in\mathcal{X}$ denotes the point along the ray at $s$, and
$\tau_{s}(r)$ is the transmittance probability at $s$, which is given by

$\mathrm{\tau}_{s}(r)\coloneqq\exp\left(-\int_{0}^{s}\sigma(s^{\prime})ds^{\prime}\right)\,.$
(2)

Using the ray rendering equation above, we can render the radiance field from
any given camera, yielding an image of the novel view. It is sufficient to
turn the camera into a proper set of rays expressed in world coordinates.

There exist various ways of implementing a radiance field, ranging from neural
networks [mildenhall2020nerf] to explicit voxel grids [ReluField_sigg_22,
SunSC22]. In this work, we opt for the latter, since it enables good rendering
quality along with faster training and inference. The explicit grid can be
queried at continuous positions via bilinear interpolation of the voxel
vertices.

Under the explicit representation, radiance fields become 4D tensors, where
the first three dimensions index a grid spanning $\mathcal{X}$, whereas the
last dimension indexes the density and color channels.

###  3.2 Generating Radiance Fields

Following recent advancements in the context of denosing-based generative
methods [ho2020denoising], we formulate our generative model of radiance
fields as a denoising diffusion probabilistic model.

Generation process. The generation (_a.k.a._ denoising) process is governed by
a discrete-time Markov chain defined on the state space $\mathcal{F}$ of all
possible pre-activated radiance fields expressed as flattened 4D tensors of
fixed size.222We consider pre-activated radiance fields, where both density
and RGB color channels span a linear space and we assume a proper activation
function will be applied at the time of rendering. This is required to have
additive noise, while preserving a valid radiance field representation. The
chain has a finite number of time steps $\\{0,\ldots,T\\}$. The denoising
process starts by sampling a state $f_{T}$ from a standard, multivariate
normal distribution $p(f_{T})\coloneqq\mathcal{N}(f_{T}|0,I)$, and generates
states $f_{t-1}$ from $f_{t}$ by leveraging reversed transition probabilities
$p_{\theta}(f_{t-1}|f_{t})$ that are Gaussian with learned parameters
$\theta$. Specifically we have

$p_{\theta}(f_{t-1}|f_{t})\coloneqq\mathcal{N}(f_{t-1}|\mu_{\theta}(f_{t},t),\Sigma_{t})\,.$
(3)

The generation process iterates up to the final state $f_{0}$, which
represents the radiance field of a 3D object generated by our method. The mean
of the Gaussian in Eq.(3) can be directly modeled with a neural network.
However, as we will see later, it is more convenient to consider the following
reparametrization of it

$\mu_{\theta}(f_{t},t)\coloneqq
a_{t}(f_{t}-b_{t}\epsilon_{\theta}(f_{t},t))\,,$ (4)

where $\epsilon_{\theta}(f_{t},t)$ is the noise that has been used to corrupt
$f_{t-1}$ predicted by _e.g_. a neural network, whereas $a_{t}$ and $b_{t}$
are pre-defined coefficients. Also the covariance $\Sigma_{t}$ takes a pre-
defined value, although it could be data-dependent. Additional details about
the value that the pre-defined variables take are given in Sec. 3.3.

Diffusion process. While the generation process works by iteratively denoising
a completely random radiance field, the diffusion process works the other way
around and iteratively corrupts samples from the distribution of 3D objects we
want to model. We introduce it because it plays a fundamental role in the
training scheme of the generation process. The diffusion process is governed
by a discrete-time Markov chain with the same state space and time bounds
mentioned in the generation process but with Gaussian transition probabilities
that are pre-defined and given by

$q(f_{t}|f_{t-1})\coloneqq\mathcal{N}(f_{t}|\sqrt{\alpha_{t}}f_{t-1},\beta_{t}I)\,,$
(5)

where $\alpha_{t}\coloneqq 1-\beta_{t}$ and $0\leq\beta_{t}\leq 1$ are
predefined coefficients implementing a schedule for the injected noise
variance. The process starts by picking $f_{0}$ from the distribution
$q(f_{0})$ of 3D object radiance fields we want to model, iteratively samples
$f_{t}$ given $f_{t-1}$ yielding a scaled and noise-corrupted version of the
latter, and stops with $f_{T}$ being typically close to completely random
depending on the implemented noise-variance schedule. By exploiting properties
of the Gaussian distribution, we can conveniently express the distribution of
$f_{t}$ conditioned on $f_{0}$ directly as a Gaussian distribution, yielding

$q(f_{t}|f_{0})=\mathcal{N}(f_{t}|\sqrt{\bar{\alpha}_{t}}f_{0},(1-\bar{\alpha}_{t})I)\,,$
(6)

where $\bar{\alpha}_{t}\coloneqq\prod_{i=1}^{t}\alpha_{i}$. This relation will
be useful to quickly generate diffused data points at arbitrary time steps.

###  3.3 Training Objective

Our training objective comprises two complementary losses: i) A loss
$L_{\mathtt{RF}}$ that penalizes the generation of radiance fields that do not
fit the data distribution, and ii) an RGB loss $L_{\mathtt{RGB}}$ geared
towards improving the quality of renderings from generated radiance fields.

Radiance field generation loss. Following [ho2020denoising], we derive the
training objective for our model starting from a variational upper-bound on
the Negative Log-Likelihood (NLL). This upper-bound requires specifying a
surrogate distribution that we refer to as $q$ because it indeed corresponds
to the distribution $q$ governing the diffusion process, establishing the
anticipated fundamental link with the generation process. We provide here some
key steps of the derivation of the bound and refer to [ho2020denoising] for
more details about the intermediate ones. By Jensen inequality, the NLL of a
data point $f_{0}\in\mathcal{F}$ can be upper-bounded by leveraging $q$ as
follows:

$-\log
p_{\theta}(f_{0})\leq\mathbb{E}_{q}\left[-\log\frac{p_{\theta}(f_{0:T})}{q(f_{1:T}|f_{0})}\right]\coloneqq
L_{\mathtt{RF}}(f_{0}|\theta)\,,$ (7)

where $f_{t_{1}:t_{2}}$ stands for $(f_{t_{1}},\ldots,f_{t_{2}})$. The loss
$L_{\mathtt{RF}}(f_{0}|\theta)$ bounding the NLL can be further decomposed
into the following sum, up to a constant independent from $\theta$

$L_{\mathtt{RF}}(f_{0}|\theta)=\sum_{t=1}^{T}L_{\mathtt{RF}}^{t}(f_{0}|\theta)+\text{const}\,.$
(8)

Here, $L_{\mathtt{RF}}^{t}(f_{0}|\theta)$ takes a simple and intuitive form if
we set $a_{t}\coloneqq\frac{1}{\sqrt{\alpha_{t}}}$ and
$b_{t}\coloneqq\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}$ in Eq.(4), and
pick
$\Sigma_{t}\coloneqq\frac{\beta_{t}^{2}}{2\alpha_{t}(1-\bar{\alpha}_{t})}I$.
Indeed, it yields

$\displaystyle L^{t}_{\mathtt{RF}}(f_{0}|\theta)$
$\displaystyle\coloneqq\mathbb{E}_{q}\left[\left\|\epsilon-\epsilon_{\theta}(f_{t},t)\right\|^{2}\right]$
$\displaystyle=\mathbb{E}_{\phi}\left[\left\|\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}f_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t)\right\|^{2}\right]\,,$

where $\phi(\epsilon)\coloneqq\mathcal{N}(\epsilon|0,I)$ is the probability
distribution of $\epsilon$, namely a normal multivariate, and the last
equality follows from Eq.(6).

Radiance field rendering loss. We complement the previous loss with an
additional RGB loss $L_{\mathtt{RGB}}(f_{0}|\theta)$, aimed at improving the
quality of renderings from generated radiance fields. Indeed, the Euclidean
metric on the representation that is implicitly used in the previous loss to
assess the quality of generated radiance fields does not necessarily ensure
the absence of artifacts once we try to render the radiance field. We define
$L_{\mathtt{RGB}}(f_{0}|\theta)$ as a sum of time-specific terms
$L^{t}_{\mathtt{RGB}}(f_{0}|\theta)$ similar to Eq.(8), yielding

$L_{\mathtt{RGB}}(f_{0}|\theta)\coloneqq\sum_{t=1}^{T}L^{t}_{\mathtt{RGB}}(f_{0}|\theta)\,.$
(9)

Given a radiance field $f\in\mathcal{F}$, a viewpoint $v$, we denote by
$R(v,f)$ the image obtained after rendering $f$ from viewpoint $v$ using
equation Eq.(1). We also denote by $\ell_{v}(f,I)$ the Euclidean distance
between the rendered images from viewpoint $v$ using radiance fields $f$ and
the ground-truth image $I_{v}$ from the view-point $v$, _i.e_.

$\ell_{v}(f,I)\coloneqq\|I_{v}-R(v,f)\|^{2}\,.$ (10)

The idea is to compare the rendering of a given radiance field $f_{0}$ sampled
from the data distribution corrupted with $t$ diffusion steps and then fully
denoised against the original ground-truth image $I_{v}$ used to obtain
$f_{0}$. In theory, this implies sampling first $f_{t}$ from $q(f_{t}|f_{0})$
and then sampling back $f_{0}$ from $p_{\theta}(f_{0}|f_{t})$. This is,
however, computationally demanding and we resort to a simpler approximation.
From the definition of $L_{\mathtt{RF}}^{t}$, the loss pushes towards having
$\epsilon\approx\epsilon_{\theta}(f_{t},t)$ from which we can derive the
approximation $\tilde{f}^{t}_{0}(\epsilon,\theta)\coloneqq
f_{0}+\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}(\epsilon-\epsilon_{\theta}(f_{t},t))$.
We can then define our rendering loss as

$L^{t}_{\mathtt{RGB}}(f_{0}|\theta)\coloneqq\omega_{t}\mathbb{E}_{\phi,\psi}\left[\ell_{v}(\tilde{f}^{t}_{0}(\epsilon,\theta),I)\right]\,,$
(11)

where the expectation is taken with respect to a prior distribution $\psi$ for
the viewpoint $v$ and $\epsilon\sim\phi(\epsilon)$. Since the approximation is
reasonable only with steps $t$ close to zero, we introduce a weight $w_{t}$
that decays as step values increase (_e.g_. we use
$\omega_{t}\coloneqq\bar{\alpha}_{t}^{2}$). We provide evidence in the
experimental section that despite being an approximation, the proposed loss
contributes to significantly improving the results.

Final loss. To summarize, the final training loss per data point $f_{0}$ is
given by the weighted combination of the radiance field generation and
rendering losses introduced before, with a small variation that enables
stochastic sampling of the step $t$ from a uniform distribution $\kappa(t)$:

$\displaystyle L(\theta)$ $\displaystyle\coloneqq
L_{\mathtt{RF}}(f_{0}|\theta)+\lambda_{\mathtt{RGB}}L_{\mathtt{RGB}}(f_{0}|\theta)$
$\displaystyle\propto\mathbb{E}_{\kappa}\left[L_{\mathtt{RF}}^{t}(f_{0}|\theta)+\lambda_{\mathtt{RGB}}L_{\mathtt{RGB}}^{t}(f_{0}|\theta)\right].$

Implementation details We implement $\epsilon_{\theta}$ as a 3D-UNet, which is
based on the 2D-UNet architecture introduced in [dhariwal2021diffusion] by
replacing 2D convolutions and attention layers with corresponding 3D
operators. For training, we uniformly sample timesteps $t=1,\ldots,T=1000$ for
all experiments with variances of the diffusion process linearly increasing
from $\beta_{1}=0.0015$ to $\beta_{T}=0.05$, and choose to weight the
rendering loss $L_{\mathtt{RGB}}^{t}$ with $w_{t}=\bar{\alpha}_{t}^{2}$. We
refer to the supplementary material for additional details.

##  4 Experiments

In this section, we evaluate the performance of our method on both
unconditional and conditional radiance field generation.

Datasets. We run experiments on the PhotoShape Chairs [photoshape2018] and on
the Amazon Berkeley Objects (ABO) Tables dataset [collins2022abo]. For
PhotoShape Chairs, we render the provided 15,576 chairs using Blender Cycles
[Blender] from 200 views on an Archimedean spiral. For ABO Tables, we use the
provided 91 renderings with 2-3 different environment map settings per object,
resulting in 1676 tables. Since both datasets do not provide radiance field
representations of 3D objects, we generate them using a voxel-based approach
at a resolution of $32^{3}$ from the multi-view renderings.

Metrics. We evaluate image quality using the Fréchet Inception Distance [FID]
(FID) and Kernel Inception Distance [KID] (KID) using
[obukhov2020torchfidelity]. For comparison of the geometrical quality, we
follow [achlioptas2018learning] and compute the Coverage Score (COV) and
Minimum Matching Distance (MMD) using Chamfer Distance (CD). While the
Coverage Score measures the diversity of the generated samples, MMD assesses
the quality of the generated samples. All metrics are evaluated at a
resolution of $128\times 128$.

###  4.1 Unconditional Radiance Field Synthesis

Comparison against state of the art. We quantitatively evaluate our approach
on the task of unconditional 3D synthesis on PhotoShape[photoshape2018] in
Tab. 1 and ABO Tables[collins2022abo] in Tab. 2. We compare against leading
methods for 3D-aware image synthesis: $\pi$-GAN [chanmonteiro2020pi-GAN] and
EG3D [Chan2022]. Both our method and GAN-based approaches use the same set of
rendered images for training. While we pre-process the rendered images to
create a radiance field representation for each of the shape samples, the GAN-
based methods are trained directly on the rendered images.

Compared to these approaches, our method yields overall better image quality
while achieving significant improvements in geometrical quality and diversity.
Fig. 3 and Fig. 4 show a qualitative comparison of our method with $\pi$-GAN
and EG3D. While EG3D achieves good image quality, it tends to produce
inaccurate shapes and view-dependent image artifacts, like adding or removing
armrests or changing the supporting structure. As the training objective (see
Sec. 3.3) is to invert the diffusion process to denoise towards detailed
volumetric representations, we observe that DiffRF reliably generates radiance
fields with fine photo-metric and geometrical details.

Contribution of the rendering loss. Tab. 1 and Tab. 2 show ablation results
where we evaluate the effects of 2D supervision on the radiance synthesis.
Removing 2D supervision (“DiffRF w/o 2D”, line 3 in the tables), as expected,
has a noticeable effect on FID, which increases by $\approx 2.3$ for
PhotoShape and by $\approx 8.8$ for ABO Tables. This shows that biasing the
noise prediction formulation from DDPM by a volumetric rendering loss leads to
higher image quality. Qualitative comparisons can be found in the appendix. We
notice a decrease in the Coverage Score that we explain by the fact that the
rendering loss guides the denoising model towards learning radiance fields
with less artifacts, thus reducing the amount of diverse but spurious shapes.

Method | FID ↓ | KID ↓ | COV $\uparrow$ | MMD ↓
---|---|---|---|---
$\pi$-GAN [chanmonteiro2020pi-GAN] | 52.71 | 13.64 | 39.92 | 7.387
EG3D [Chan2022] | 16.54 | 8.412 | 47.55 | 5.619
DiffRF w/o 2D | 18.27 | 9.263 | 59.20 | 4.543
DiffRF | 15.95 | 7.935 | 58.93 | 4.416

Table 1: Quantitative comparison of unconditional generation on the PhotoShape
Chairs [photoshape2018] dataset. Our method achieves a better image and
geometric quality than state-of-the-art GAN-based approaches. The additional
2D rendering loss improves image quality as indicated by the drop in quality
without it. MMD and KID scores are multiplied by $10^{3}$.

Method | FID ↓ | KID ↓ | COV $\uparrow$ | MMD ↓
---|---|---|---|---
$\pi$-GAN [chanmonteiro2020pi-GAN] | 41.67 | 13.81 | 44.23 | 10.92
EG3D [Chan2022] | 31.18 | 11.67 | 48.15 | 9.327
DiffRF w/o 2D | 35.89 | 13.94 | 63.46 | 8.013
DiffRF | 27.06 | 10.03 | 61.54 | 7.610

Table 2: Quantitative comparison of unconditional generation on the ABO Tables
[collins2022abo] dataset. Our method achieves a better image and geometric
quality than state-of-the-art GAN-based approaches, with the 2D rendering loss
being important. MMD and KID scores are multiplied by $10^{3}$.

Masking: | 20% | 40% | 60% | 80% | Avg
---|---|---|---|---|---
mPSNR$\uparrow$ | EG3D | 23.71 | 24.86 | 24.92 | 25.79 | 24.82
DiffRF | 24.85 | 26.66 | 28.23 | 30.38 | 27.53
FID$\downarrow$ | EG3D | 25.91 | 29.41 | 33.06 | 34.31 | 30.67
DiffRF | 22.36 | 27.74 | 31.16 | 29.84 | 27.78

Table 3: Quantitative evaluation on the task of radiance field completion at
different levels of masking. EG3D struggles to faithfully reconstruct non-
masked regions of the original sample, while our method maintains the non-
masked regions and synthesises coherent completions for the masked regions.
Figure 3:  Qualitative comparison between $\pi$-GAN [chanmonteiro2020pi-GAN],
EG3D [Chan2022], and our method on PhotoShape Chairs [photoshape2018]. Our
approach leads to diverse, geometrically accurate models that allow for high-
quality renderings.  Figure 4: Qualitative comparison between $\pi$-GAN
[chanmonteiro2020pi-GAN], EG3D [Chan2022], and our method on ABO Tables
[collins2022abo]. Our approach generates high quality and diverse samples with
accurate geometry.

###  4.2 Conditional Generation

GANs need to be trained in order to be conditioned on a particular task, while
diffusion models can be effectively conditioned at test-time
[dhariwal2021diffusion, song2020score, Lugmayr_2022_CVPR]. We leverage this
property for the novel task of masked radiance field completion.

Masked Radiance Field Completion. Shape completion and image inpainting are
well-studied tasks [pointflow, zhou20213d, yu2018generative,
suvorov2021resolution] aiming to fill missing regions within a geometrical
representation or in an image, respectively. We propose to combine both in the
novel task of masked radiance field completion: Given a radiance field and a
3D mask, synthesize a completion of the masked region that harmonizes with the
non-masked region. Inspired by RePaint [Lugmayr_2022_CVPR], we perform
conditional completion by gradually guiding the unconditional sampling process
in the known region to the input $f^{in}$

$\displaystyle f_{0}^{t-1}$
$\displaystyle=\sqrt{\bar{\alpha}_{t}}(m\odot\tilde{f}^{t}_{0}+(1-m)\odot
f^{in})$ $\displaystyle f_{t-1}$
$\displaystyle\sim\mathcal{N}(f_{0}^{t-1},(1-\bar{\alpha}_{t})I)\,,$

where $m$ is a binary mask applied to the input (light blue in Fig. 6) and
$\odot$ denotes element-wise multiplication on the voxel grid.

A quantitative analysis of the masking performance is shown in Tab. 3, where
we compare our method against EG3D at different levels of masking. At each
level, we randomly mask 200 samples and evaluate the completion performance in
terms of FID (by rendering from 10 random views), as well as the photo-metric
accuracy of unmasked regions (mPSNR). For EG3D, we use masked GAN inversion,
where we perform global latent optimization (GLO) to minimize the photo-metric
error on the re-projected, unmasked regions. We notice that, due to the single
latent code representation, EG3D struggles to faithfully reconstruct the
unmasked region of the input sample, and regularization is needed to not
corrupt the overall representation. Fig. 5 further shows a qualitative
comparison of the masking performance against EG3D.

Figure 5:  Qualitative completion of masked chairs from PhotoShape
[photoshape2018]. DiffRF shows more diverse proposals compared to EG3D, while
also maintaining the original non-masked regions.  Figure 6:  Masked radiance
field completion. For a trained denoiser $\mathcal{D}$ and a given a radiance
field with a 3D masked region (shown in blue), completion can be obtained by
iteratively fusing the non-masked input region with the estimated denoisings
$\tilde{f}^{t}_{0}$ and applying the diffusion process on the resulting
$f_{0}^{t-1}$.

Image-to-Volume Synthesis. DiffRF can be used to obtain 3D radiance fields
from single view images by steering the sampling process using volumetric
rendering. For this, we adopt the Classifer Guidance formulation from
[dhariwal2021diffusion] to guide the denoising process towards minimizing the
rendering error against a posed RGB image with corresponding object mask
(obtained, for example, with off-the-shelf segmentation networks). Fig. 7
shows qualitative results for this single-image reconstruction task on chairs
from ScanNet [dai2017scannet]. Furthermore, we show results of our model
conditioned on CLIP-embeddings [radford2021learning] in the appendix.

Figure 7:  Qualitative synthesis results of single-view reconstruction on
ScanNet [dai2017scannet]. Given a single posed image and its foreground mask,
our method can recover meaningful proposals for the radiance fields describing
the objects (shown in separate rows).

###  4.3 Limitations

While our method shows promising results on the tasks of conditional and
unconditional radiance field synthesis, several limitations remain. Compared
to GAN-based approaches, our radiance fields-based approach requires a
sufficient number of posed views in order to generate good training samples
and suffers from lower sampling times. In this context, it would be
interesting to explore leveraging faster sampling methods [kong2021fast].
Finally, our model is constrained in the maximum grid resolution by training-
time memory limitations. These could be addressed by exploring adaptive
[takikawa2021nglod, Wang-2017-ocnn] or sparse grid structures
[fridovich2022plenoxels, takikawa2021nglod, schwarzvoxgraf] as well as
factorized neural fields representations [chen2022tensorf, wang2022rodin].

##  5 Conclusions

We introduce DiffRF – a novel approach for 3D radiance field synthesis based
on denoising diffusion probabilistic models. To the best of our knowledge,
DiffRF is the first generative diffusion-based method to operate directly on
volumetric radiance fields. Our model learns multi-view consistent priors from
collections of posed images, enabling free-view image synthesis and accurate
shape generation. We evaluated DiffRF on several object classes, comparing its
performance against state-of-the-art GAN-based approaches, and demonstrating
its effectiveness in both conditional and unconditional 3D generation tasks.

## Acknowledgements

This work was done during Norman’s and Yawar’s internships at Meta Reality
Labs Zurich as well as at TUM, funded by a Meta SRA. Matthias Nießner was also
supported by the ERC Starting Grant Scan2CAD (804724).