# Goal-Oriented State Information Compression for Linear Dynamical System
Control

Li Wang12, Chao Zhang12, Samson Lasaulce13, Lina Bariah14, and Merouane
Debbah1  16G Research Center, Khalifa University, Abu Dhabi, UAE  2Central
South University, Changsha, China  3University of Lorraine, Nancy, France
4Open Innovation AI, Abu Dhabi, UAE

###### Abstract

In this paper, we consider controlled linear dynamical systems in which the
controller has only access to a compressed version of the system state. The
technical problem we investigate is that of allocating compression resources
over time such that the control performance degradation induced by compression
is minimized. This can be formulated as an optimization problem to find the
optimal resource allocation policy. Under mild assumptions, this optimization
problem can be proved to have the same well-known structure as in [1],
allowing the optimal resource allocation policy to be determined in closed-
form. The obtained insights behind the optimal policy provide clear guidelines
on the issue of ”when to communicate” and ”how to communicate” in dynamical
systems with restricted communication resources. The obtained simulation
results confirm the efficiency of the proposed allocation policy and
illustrate the gain over the widely used uniform rate allocation policy.

###### Index Terms:

goal-oriented compression; networked linear dynamical system; resource
allocation; linear quadratic regulator.

##  I Introduction

The rapid growth of data-driven technologies and widespread use of smart IoT
devices has led to a massive increase of data volumes, challenging future
wireless networks to enhance transmission rates, energy efficiency, latency,
and reliability. As network capacity expansion lags behind data growth,
revisiting data compression techniques becomes crucial for efficient data
transmission and resource management.

Recent developments in semantic and goal-oriented compression methods have
become key to addressing increasing data volumes [2][3]. Unlike traditional
compression, which focuses on minimal data loss, these methods align
compression with specific tasks, maintaining quality and essential features
for objectives such as real-time decision-making [4], efficient resource
allocation [5], and inference [6], among others. For instance, [7] considers
the final use of the data to the system task modeled by an optimization
problem and utilizes high-resolution quantization theory to establish the
effectiveness of a goal-oriented quantization scheme. The authors of [8]
proposed a synonymous mapping to describe the correlation between the semantic
information and the original source data, reducing coding length in both lossy
and lossless scenarios. Given the abstract nature of semantic information,
deep learning techniques are extensively employed to extract relevant semantic
content and key task features, employing methods such as transformer-based
joint-source encoder-decoders [9] and generative models [10]. Most existing
studies focus on static systems with independent and identically distributed
(i.i.d.) observation processes, emphasizing data meaning preservation over
managing dynamic tasks, rather than handling tasks that involve dynamic or
time-varying random processes. As the next-generation communication systems
are expected to remotely control a wide range of dynamic systems, including
cars, robots, and drones, optimizing the use of communication resources is
essential to ensure reliable and efficient wireless connectivity for these
advanced applications.

In dynamical systems, actions have long-lasting effects on future states,
often sacrificing immediate rewards for long-term benefits. The information to
be compressed, including system states and observations, typically exhibits
memory and can be modeled with linear recursive equations, Markov decision
processes (MDP), or more complex structures. This feature highlights the
importance of temporal correlation in the development of control policies.
Various studies have addressed the dynamic aspects of systems; for instance,
[11] proposed adaptive power control policies in linear systems for vehicle
platooning, while [12] introduces a threshold-based communication scheme that
leverages a waiting time policy for energy-efficient wireless control of
nonlinear systems. In MDP settings, [13] develops a goal-oriented sampling
policy to minimize actuation errors in real-time tracking under resource
limitations, and [14] explores the integration of quantization and control in
bit-budgeted communications. Despite these advancements, the impact of
compression noise on dynamical systems remains under-explored, and addressing
this gap is a primary focus of our current research.

In this paper, we propose a goal-oriented compression framework for optimal
resource allocation, focusing on the effects of compression noise. We analyze
the noise impact using the linear quadratic regulator (LQR) problem, employing
a non-i.i.d. system state modeled by a recursive equation. Utilizing Shannon’s
rate-distortion theory, we establish a relationship between noise variance and
transmission rate, deriving a closed-form expression for the optimal rate that
decreases over time, indicating a higher initial resource allocation in an
invariant system. The presented simulation results further validate the
efficiency of our proposed method.

##  II Problem formulation

The rate-limited control setting discussed in this article features an LQR
system with a state vector $\mathbf{x}_{t}\in\mathbb{R}^{n}$ and a
communication link between an observer and the controller, as depicted in Fig.
1.

Figure 1: LQR setting with a communication link (the dashed line) from the
observer to the controller.

To evaluate performance degradation due to limited transmission rates, it’s
crucial to analyze state deviations from imperfect communication. We compare a
control system with perfect communication to one with rate-limited
communication, denoting their state-control pairs as
$(\mathbf{x}^{(p)}_{\tau},\mathbf{u}^{(p)}_{\tau})$ and
$(\mathbf{x}^{(c)}_{\tau},\mathbf{u}^{(c)}_{\tau})$, respectively.

###  II-A Perfect transmission

The linear dynamical system with perfect communication link is described by:

$\mathbf{x}^{(p)}_{t+1}=\mathbf{A}\mathbf{x}^{(p)}_{t}+\mathbf{B}\mathbf{u}^{(p)}_{t}+\mathbf{z}_{t},$
(1)

where $\mathbf{A}\in\mathbb{R}^{n\times n}$ and
$\mathbf{B}\in\mathbb{R}^{n\times m}$ are system matrices,
$\mathbf{x}^{(p)}_{t}\in\mathbb{R}^{n}$ is the state,
$\mathbf{u}^{(p)}_{t}\in\mathbb{R}^{m}$ is the control input and
$\mathbf{z}_{0},\mathbf{z}_{1},\dots$ are i.i.d. disturbances with zero mean
and covariance $\Sigma_{z}\in\mathbb{S}^{n}_{++}$, where $\mathbb{S}^{n}_{++}$
denotes the set of all positive definite $n\times n$ matrices.

For a finite time system, the design of the control policy in a typical
control system with perfect transmission aims to minimize the average-stage
LQR cost upon reaching the horizon $T$:

$\min_{\mathbf{u}^{(p)}_{0},\mathbf{u}^{(p)}_{1},\dots}J^{(p)}\triangleq\frac{1}{T}\mathbb{E}\left[\sum_{t=0}^{T-1}\mathbf{x}^{(p)^{\prime}}_{t}\mathbf{Q}\mathbf{x}^{(p)}_{t}+\mathbf{u}^{(p)^{\prime}}_{t}\mathbf{D}\mathbf{u}^{(p)}_{t}\right],$
(2)

where $x^{\prime}$ represents the transpose, and
$\mathbf{Q}\in\mathbb{S}^{n}_{++}$ and $\mathbf{D}\in\mathbb{S}^{n}_{++}$ are
cost matrices. The expectation is calculated with respect to the disturbance
process $\mathbf{z}_{t}$. It has been shown that to solve (2), the optimal
strategy is to use control actions which are proportional to the current
system state as follows

$\mathbf{u}^{(p)}_{t}=\mathbf{F}_{t}\mathbf{x}^{(p)}_{t},$ (3)

where $\mathbf{F}_{t}\in\mathbb{R}^{m\times n}$ is the control matrix and can
be computed as in [15]. By inserting $\mathbf{u}^{(p)}_{t}$ into (2),
$\mathbf{F}_{t},~{}0\leq t\leq T$ are estimated by:

$\min_{\mathbf{F}_{0},\mathbf{F}_{1},\dots}J^{(p)}\triangleq\frac{1}{T}\mathbb{E}\left[\sum_{t=0}^{T-1}\mathbf{x}^{(p)^{\prime}}_{t}(\mathbf{Q}+\mathbf{F}_{t}^{\prime}\mathbf{D}\mathbf{F}_{t})\mathbf{x}^{(p)}_{t}\right],$
(4)

where $T$ denotes the transpose of matrix.

###  II-B Rate-limited transmission

In a rate-limited system, the observed system state is transmitted
imperfectly. We denote $\mathbf{n}_{t}\in\mathbb{R}^{n}$ as the deviation
between the real system state and the compressed system state as follows:

$\hat{\mathbf{x}}^{(c)}_{t}=\mathbf{x}^{(c)}_{t}+\mathbf{n}_{t}^{(c)},~{}~{}\forall~{}0\leq
t\leq T,$ (5)

where

$\mathbf{n}_{t}^{(c)}\sim p(R_{t}),~{}~{}\forall~{}0\leq t\leq T.$ (6)

Here $p(R_{t})$ represents a probability density function (p.d.f.) that is a
function of the transmission rate $R_{t}$ (rate allocated at time $t$). A
higher transmission rate $R_{t}$ often yields a lower value of
$\mathbf{n}_{t}$, leading to a smaller variance of the distribution
$p(R_{t})$. With the compression noise, biased control inputs
$\mathbf{u}^{(c)}_{t}$ and biased respective states $\mathbf{x}^{(c)}_{t}$ are
derived, according to the dynamic process, as follows

$\displaystyle\mathbf{u}^{(c)}_{t}$
$\displaystyle=\mathbf{F}_{t}\hat{\mathbf{x}}^{(c)}_{t},$
$\displaystyle\mathbf{x}^{(c)}_{t+1}$
$\displaystyle=\mathbf{A}\mathbf{x}^{(c)}_{t}+\mathbf{B}\mathbf{u}^{(c)}_{t}+\mathbf{z}_{t},$
(7)

where $\mathbf{u}^{(c)}_{t}=\mathbf{F}_{t}\hat{\mathbf{x}}^{(c)}_{t}$, and
$\hat{\mathbf{x}}^{(c)}_{t}$ is the compressed observation of the state at
time $t$, as defined in (5). The average-stage LQR cost then become

$J^{(c)}(R_{0},R_{1},\dots,R_{T})\triangleq\frac{1}{T}\mathbb{E}\left[\sum_{t=0}^{T-1}\mathbf{x}^{(c)^{\prime}}_{t}\mathbf{Q}\mathbf{x}^{(c)}_{t}+\mathbf{u}^{(c)^{\prime}}_{t}\mathbf{D}\mathbf{u}^{(c)}_{t}\right]$
(8)

It can be noticed from 8 that the compression noise added at time $t$ will
have influence on the LQG cost. It is also worthy to note that the noise
intensity at time $t$ is influenced by the allocated rate at that time. When
the allocated rate is unlimited, the compression noise is negligible. However,
in a linear dynamical system with rate-limited communication, the rate
allocated at each time is limited by the available resources. In this work, we
consider the scenario where the total rate budget is limited, i.e., the sum of
rate should be smaller than a threshold, namely
$\sum_{t=0}^{T}R_{t}<R_{\mathrm{sum}}$, where $R_{t},~{}0\leq t\leq T$ is the
allocated rate at time $t$, and $R_{\mathrm{sum}}$ represent the upper bound
of the overall rate.

The goal of the resource allocation scheme is to minimize the LQR costs caused
by the compression process, while ensuring that the transmission rate is
within the available rate budget. More precisely, the rate allocation scheme
can be formulated as the following constrained optimization problem

$\displaystyle\min_{(R_{0},R_{1},\dots,R_{T})}J(R_{0},R_{1},\dots,R_{T})\triangleq
J^{(c)}-J^{(p)},$ $\displaystyle\mathrm{s.t.}~{}~{}\sum_{t=0}^{T}R_{t}\leq
R_{\mathrm{sum}}$ (9)

##  III Analysis of the rate allocation problem

For notational convenience and calculation simplification, we study the
standard setup of discrete LQR problems as in Assumption 1. However, we would
like to claim that the way to derive the rate allocation policy for vector
state systems is quite similar to scalar state systems.

######  Assumption 1.

The system state and control action are both scalars.

According to this assumption, we rewrite the perfect communication system as
follows

$\displaystyle x_{t+1}^{(p)}$ $\displaystyle=Ax_{t}^{(p)}+Bu_{t}^{(p)}+z_{t},$
$\displaystyle u_{t}^{(p)}$ $\displaystyle=F_{t}\cdot x_{t}^{(p)},$ (10)

Assuming that the controller coefficients are optimized and fixed, then a
system with a rate-limited communication can be given by

$\displaystyle x_{t+1}^{(c)}$ $\displaystyle=Ax_{t}^{(c)}+Bu_{t}^{(c)}+z_{t},$
$\displaystyle u_{t}^{(c)}$ $\displaystyle=F_{t}\cdot\hat{x}_{t}^{(c)},$
$\displaystyle\hat{x}_{t}^{(c)}$ $\displaystyle=x_{t}^{(c)}+n_{t}^{(c)},$ (11)

In order to quantify the performance degradation caused by rate-limited
communications in LQR problems, it is essential to evaluate the biases between
system states with and without noise, and this is the motivation behind the
following proposition.

######  Proposition 1.

For a rate-limited invariant dynamical system, the state at time $t+1$ can be
derived from the state in a perfect-transmission system as

$x_{t+1}^{(c)}=x_{t+1}^{(p)}+\sum_{m=0}^{t}\left(\prod_{i=0}^{t-m}(A*\mathrm{sgn}(i)+BF_{m+i})\right)n_{m}^{(c)},$
(12)

for $0\leq t\leq T$, where $x_{t+1}^{(p)}$ and $x_{t+1}^{(c)}$ represent
respectively the state in a perfect-transmitted system, and the state with
compressed transmission, at time $t+1$. $\mathrm{sgn(\cdot)}$ is the sign
function.

###### Proof.

From (10) and (11), we can derive

$\displaystyle x_{t+1}^{(p)}$ $\displaystyle=(A+BF_{t})x_{t}^{(p)}+z_{t}$ (13)
$\displaystyle x_{t+1}^{(c)}$
$\displaystyle=(A+BF_{t})x_{t}^{(c)}+z_{t}+(BF_{t})n_{t}^{(c)},$ (14)

for $0\leq t\leq T$. By subtracting (13) from (14), the auto-regressive
function can be derived as

$x_{t+1}^{(c)}-x_{t+1}^{(p)}=(A+BF_{t})(x_{t}^{(c)}-x_{t}^{(p)})+(BF_{t})n_{t}^{(c)}.$
(15)

Note that the first state is not influenced by a compressed control policy,
i.e., $x_{0}^{(c)}=x_{0}^{(p)}$, thus we can obtain

$\displaystyle
x_{t+1}^{(c)}-x_{t+1}^{(p)}=\sum_{m=0}^{t}\left(\prod_{i=0}^{t-m}(A*\mathrm{sgn}(i)+BF_{m+i})\right)n_{m}^{(c)}.$

By applying transposition of terms, we obtain

$x_{t+1}^{(c)}=x_{t+1}^{(p)}+\sum_{m=0}^{t}\left(\prod_{i=0}^{t-m}(A*\mathrm{sgn}(i)+BF_{m+i})\right)n_{m}^{(c)}.$
(16)

∎

While the differences in system states can be explicitly described, assessing
the impact of compression noise on system costs is challenging due to the
complex nature of noise. To address this, we make specific assumptions to
simplify the theoretical analysis.

######  Assumption 2.

The compression noise is assumed to be independent of the signals and follows
the Gaussian distribution, namely,

$n_{t}^{(c)}\sim\mathcal{N}(0,c^{2}2^{-2R_{t}}),$ (17)

where $R_{t}$ is the rate allocated to transmit the state at time slot $t$,
and $c$ is a compression-related constant.

Based on this assumption, the system state bias could be further derived in
the following proposition.

######  Proposition 2.

When the compression noise satisfies Assumption 2, the system state can be
expressed as

$x_{t+1}^{(c)}=x_{t+1}^{(p)}+\bar{n}_{t+1}^{(c)},$ (18)

where

$\bar{n}_{t+1}^{(c)}\sim\mathcal{N}(0,\sigma_{t+1}^{2})$ (19)

and

$\sigma_{t+1}^{2}=c^{2}\sum_{m=0}^{t}\left(\prod_{i=0}^{t-m}(A*\mathrm{sgn}(i)+BF_{m+i})\right)^{2}2^{-2R_{m}}$
(20)

###### Proof.

Based on Eq.16 we have

$x_{t+1}^{(c)}=x_{t+1}^{(p)}+\bar{n}_{t+1}^{(c)}$

where

$\bar{n}_{t+1}^{(c)}=\sum_{m=0}^{t}\left(\prod_{i=0}^{t-m}(A*\mathrm{sgn}(i)+BF_{m+i})\right)n_{m}^{(c)}.$

Knowing that the sum of independent Gaussian variables still follows Gaussian
distribution with the expectation and variance equal to the sum of expectation
and variance of each variable, and
$n_{m}^{(c)}\sim\mathcal{N}(0,c^{2}2^{-2R_{m}})$ according to Assumption 17,
straightforward proving Proposition 20.

∎

As system state bias induced by compression noise can be approximated by
additive Gaussian noise, the rate allocation problem defined by (9) can be
proved equivalent to a convex optimization problem, shown in Proposition 21.

######  Proposition 3.

Based on Assumption 1 and Assumption 2, the original rate allocation problem
defined by (9) can be equivalently solved by the following constrained
optimization problem:

$\displaystyle\min_{(R_{0},R_{1},\dots,R_{T})}\sum_{t=0}^{T}a_{t}2^{-2R_{t}},$
$\displaystyle\mathrm{s.t.}~{}~{}\sum_{t=0}^{T}R_{t}\leq R_{\text{sum}},$

where

$a_{t}=c^{2}\sum_{m=t+1}^{T}(Q+F_{m}^{2}D)\left(\prod_{i=0}^{m-1-t}(A*\mathrm{sgn}(i)+BF_{t+i})^{2}\right)+F_{t}^{2}Dc^{2}.$
(21)

###### Proof.

The optimality loss at time slot $t$ can be expressed as

$\displaystyle\quad\,\,\mathbb{E}[f_{t}(x_{t}^{(c)},u_{t}^{(c)})-f_{t}(x_{t}^{(p)},u_{t}^{(p)})]$
$\displaystyle=\mathbb{E}[Q(2x_{t}^{(p)}+\bar{n}_{t}^{(c)})\bar{n}_{t}^{(c)}+F_{t}^{2}D(\bar{n}_{t}^{(c)}+n_{t}^{(c)})(2x_{t}^{(p)}+\bar{n}_{t}^{(c)}+n_{t}^{(c)})]$
$\displaystyle=Q\sigma^{2}_{t}+F_{t}^{2}D(\sigma_{t}^{2}+c^{2}2^{-2R_{t}})$
$\displaystyle=(Q+F_{t}^{2}D)c^{2}\sum_{m=0}^{t-1}\left(\prod_{i=0}^{t-1-m}(A*\mathrm{sgn}(i)+BF_{m+i})^{2}\right)2^{-2R_{m}}$
$\displaystyle\quad\quad+F_{t}^{2}Dc^{2}2^{-2R_{t}}$

For the bit allocation problem, which can be formulated as

$\min\sum_{t=0}^{T}\mathbb{E}[f_{t}(x_{t}^{(c)},u_{t}^{(c)})-f_{t}(x_{t}^{(p)},u_{t}^{(p)})]$

subject to:

$\sum_{t=0}^{T}R_{t}\leq R_{\text{sum}}$

the objective function can be rewritten as

$a_{0}2^{-2R_{0}}+a_{1}2^{-2R_{1}}+a_{2}2^{-2R_{2}}+\ldots+a_{T}2^{-2R_{T}}$

where

$a_{t}=c^{2}\sum_{m=t+1}^{T}(Q+F_{m}^{2}D)\left(\prod_{i=0}^{m-1-t}(A*\mathrm{sgn}(i)+BF_{t+i})^{2}\right)+F_{t}^{2}Dc^{2}$

for $1<t<T$, and $a_{T}=0$. ∎

The solution of the optimization problem can be obtained easily by using
Lagrangian multiplier approach, and can be written as

$R_{k}^{*}=\frac{1}{2}\log\frac{a_{k}}{\sqrt[T]{\prod_{i=0}^{T-1}a_{i}}}+\frac{R_{\text{sum}}}{T}$

for $0\leq k<T$, and $R_{T}^{*}=0$.

######  Corollary 1.

In a system with an invariant controller $F_{0}=F_{1}=F_{2}=\cdots=F$, the
optimal rate with a given overall budget $R_{t}^{*}$ is a decreasing function
of $t$, namely

$R_{0}^{*}>R_{1}^{*}>R_{2}^{*}>\ldots>R_{T}^{*},$ (22)

indicating that more transmission resource should be allocated to early stages
in this case.

This can be explained by the fact that the compression noise at early stages
will be accumulated in subsequent stages, thus having smaller error in early
stages yields a better performance.

##  IV Numerical analysis

In this section, we present simulation results to show the performance of the
proposed rate allocation scheme in dynamic wireless systems.

###  IV-A Evaluation metric

A relative LQR cost, represented by $J_{RCost}$, is used to evaluate the
performance of the proposed rate-allocation scheme, where

$J_{RCost}=(J^{(c)}-J^{(p)})/J^{(p)},$ (23)

given that $J^{(p)}$ and $J^{(c)}$ are respectively the LQR cost of perfect-
transmitted system and networked control system, defined in Eq. (2) and Eq.
(8). The goal of the rate-allocation scheme is to mitigate the additional cost
caused by the transmission rate constraint, or equivalently, to minimize the
relative LQR cost.

###  IV-B Time-invariant systems

The considered system parameters are defined as follows: $B=1$, $Q=2$, and
$D=5$, with $A$ varying between $[0,3]$. For each case, the optimal controller
$F_{t},0\leq t\leq T$ is obtained according to the standard formula [15]. This
study models a 11-stage finite horizon system starting from $x_{0}=100$.
Gaussian noise is added to each state $x_{t}$, with variance determined by
(17). The relative costs are shown in Figs.2, while Fig. 3 illustrates the
optimized rate allocation scheme under various settings of $A$. We observe
that with a lower $A$ value, the performance of the optimized rate-allocation
scheme approaches the constant rate scheme, resulting in a smaller difference
in relative LQR costs between the proposed and constant-cost schemes.
Conversely, with a higher $A$ value, the superiority of the optimal rate
allocation scheme becomes more pronounced, leading to significantly lower LQR
costs.

This finding aligns with the following: when $A$ is small, previous states
have less influence on the current state, thereby increasing the impact of the
current controller. As a result, the noise from the earlier state
transmissions has less effect on the overall cost, making the transmission
noise for each state equally critical. Conversely, when previous states
significantly influence the current state (larger $A$), more resources are
dedicated to transmitting earlier states to ensure accuracy and minimize cost
impacts.

Figure 2: Relative LQR costs (23), The effectiveness of the proposed optimal
rate allocation scheme is significantly enhanced when the dynamic process is
more influenced by state transitions than by controller inputs (i.e., when the
value of $A$ is higher), outperforming the constant-rate scheme more
noticeably. Figure 3: Optimal rate allocation for invariant linear dynamic
systems with 11 stages, defined with different $(A,B)$ pairs. It demonstrates
that when the dynamical process is more influenced by state transitions than
by controller inputs (i.e., when A is larger), more transmission resources
should be allocated to the earlier states.

###  IV-C Time-variant dynamical systems

Dynamical systems often experience changes due to external disturbances. For
example, a control system might alter when faced with environmental
perturbations. This paper shows that an optimized rate allocation scheme can
also adapt to changes in a dynamical system. Here we show the optimized
allocated rate for a variant dynamical system with a change characterized by a
sudden shift in the system coefficient $A$. Specifically, due to the
computational complexity involved, we consider a system with 4 stages,
undergoing a coefficient change at time $t=3$, switching from $A_{1}$ to
$A_{2}$. Through exhaustive search, the optimal rate allocation for this
variable system is determined, revealing that rate allocation should increase
when there is a jump in $A$. More over, the allocated rate increases in the
time before the jump appears. This adaptation is illustrated in Fig. 4.

Figure 4: Optimal rate allocation for invariant linear dynamic systems with 4
stages, where $A$ switches from $A_{1}$ to $A_{2}$ at time $t=3$. It indicates
that when the influence of state information grows (i.e., when the parameter
$A$ increases) in a variant dynamic system, more resources should be allocated
to the stages near the point of change.

##  V Conclusion

We study goal-oriented compression problems in a linear dynamical system with
rate-limited communication resources by considering the long-lasting effect of
compression noise on future system states. Our primary objective is to find
the optimal transmission rate to reduce the performance degradation induced by
compression noise with a given budget. The proposed rate allocation strategy
indicates that it is necessary to communicate more in the early stages due to
the noise accumulation effect, and also at time slots where there is a
noticeable shift of the system dynamics, to mitigate the control action
deviation caused by these state changes.

## References

  * [1] P. P. Vaidyanathan. ”Multirate systems and filter banks”, Pearson Education India, 2006.
  * [2] C. Zhang et al.,“Goal-Oriented Communications for the IoT and Application to Data Compression,” in IEEE Internet of Things Magazine, vol. 5, no. 4, pp. 58-63, December 2022.
  * [3] D. Gündüz et al., “Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications,” in IEEE Journal on Selected Areas in Communications, vol. 41, no. 1, pp. 5-41, Jan. 2023.
  * [4] C. Zhang et al., “Decision-making oriented clustering: Application to pricing and power consumption scheduling”, Appl. Energy, vol. 297, Sep. 2021.
  * [5] H. Zhang et al., “DRL-Driven Dynamic Resource Allocation for Task-Oriented Semantic Communication,” in IEEE Transactions on Communications, vol. 71, no. 7, pp. 3992-4004, July 2023.
  * [6] J. Chen et al., “Neuromorphic Wireless Cognition: Event-Driven Semantic Communications for Remote Inference,” in IEEE Transactions on Cognitive Communications and Networking, vol. 9, no. 2, pp. 252-265, April 2023.
  * [7] H. Zou et al., “Goal-Oriented Quantization: Analysis, Design, and Application to Resource Allocation,” in IEEE Journal on Selected Areas in Communications, vol. 41, no. 1, pp. 42-54, Jan. 2023.
  * [8] K. Niu, and P. Zhang. “A mathematical theory of semantic communication.” arXiv preprint arXiv:2401.13387, 2024.
  * [9] H. Xie et al., “Deep Learning Enabled Semantic Communication Systems,” in IEEE Transactions on Signal Processing, vol. 69, pp. 2663-2675, 2021.
  * [10] E. Grassucci et al., “Generative semantic communication: Diffusion models beyond bit recovery.” arXiv preprint arXiv:2306.04321, 2023.
  * [11] X. Kui et al., “Towards Task-Oriented Communication Strategies for Platooning by Deep Reinforcement Learning,” 2023 21st International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), Singapore, Singapore, 2023, pp. 1-7.
  * [12] V. S. Varma et al., “Transmission Power Policies for Energy-Efficient Wireless Control of Nonlinear Systems,” in IEEE Transactions on Automatic Control, vol. 68, no. 6, pp. 3362-3376, June 2023.
  * [13] M. Salimnejad et al., “Real-time Reconstruction of Markov Sources and Remote Actuation over Wireless Channels,” in IEEE Transactions on Communications, 2024.
  * [14] A. Mostaani et al., “Task-Effective Compression of Observations for the Centralized Control of a Multiagent System Over Bit-Budgeted Channels,” in IEEE Internet of Things Journal, vol. 11, no. 4, pp. 6131-6143, 15 Feb.15, 2024.
  * [15] A. Shaiju and I. Petersen, “Formulas for discrete time LQR, LQG LEQG and minimax LQG optimal control,” in Proc. Int. Federation Autom. Control, 2008, pp. 8773–8778.


* Weinland et al. [2006] Weinland, D., Ronfard, R., Boyer, E.: Free viewpoint action recognition using motion history volumes. Computer Vision and Image Understanding 104, 249–257 (2006). Special Issue on Modeling People: Vision-based understanding of a person’s shape, appearance, movement and behaviour


# Ordinary Differential Equation-based CNN for Channel Extrapolation over RIS-
assisted Communication

Meng Xu, Shun Zhang, _Senior Member, IEEE_ , Caijun Zhong, _Senior Member,
IEEE_ , Jianpeng Ma, _Member, IEEE_ , Octavia A. Dobre, _Fellow, IEEE_ M. Xu,
S. Zhang and J. Ma are with the State Key Laboratory of Integrated Services
Networks, Xidian University, Xi an 710071, P. R. China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>jpmaxdu@gmail.com).C. Zhong is with the College of Information Science and
Electronic Engineering, Zhejiang University, Hangzhou 310027, P. R. China
(e-mail: caijunzhong@zju.edu.cn).O. A. Dobre is with Faculty of Engineering
and Applied Science, Memorial University, St. John’s NL AIC-5S7, Canada
(e-mail: odobre@mun.ca).

###### Abstract

The reconfigurable intelligent surface (RIS) is considered as a promising new
technology for reconfiguring wireless communication environments. To acquire
the channel information accurately and efficiently, we only turn on a fraction
of all the RIS elements, formulate a sub-sampled RIS channel, and design a
deep learning based scheme to extrapolate the full channel information from
the partial one. Specifically, inspired by the ordinary differential equation
(ODE), we set up connections between different data layers in a convolutional
neural network (CNN) and improve its structure. Simulation results are
provided to demonstrate that our proposed ODE-based CNN structure can achieve
faster convergence speed and better solution than the cascaded CNN.

###### Index Terms:

Convolutional neural network, RIS, channel extrapolation, ordinary
differential equation, sub-sample.

##  I Introduction

With the increasing demands for communication services, the number of
connected devices continues to augment exponentially and new, blossoming
service requirements pose more constraints on the network. At the same time,
the increased power consumption and hardware cost remain key issues [1]. A
recent technological breakthrough that holds the potential to overcome these
technological bottlenecks is reconfigurable intelligent surface (RIS). It can
be coated on any environmental object in a cost-effective manner, thereby
facilitating the large scale deployment. Due to the physical characteristics
of RIS, it can reflect incident electromagnetic waves, and adjust their
amplitude and phase in a controlled manner. In other words, RIS can manipulate
the communication environment in an intelligent way. Furthermore, the
reflection elements of RIS usually work in a passive state, which makes RIS
have low power consumption [2]. Hence, RIS is considered as a promising
technology and attracts more and more attention.

Similar to the case of other communication systems [3], the acquisition of
channel state information (CSI) is an important problem and has become a hot
research topic in RIS-assisted communication systems. In [4], Ardah _et al._
designed a two-stage channel estimation framework with high resolution. In
[5], the atomic norm minimization was resorted to implement the channel
estimation over RIS-aided MIMO system in the millimeter wave frequency band.

All of the above works depend on the hypothetical statistical model. However,
in the actual communication scenario, the radio scattering conditions change
rapidly with time and are very complicated. This makes the traditional methods
have some limitations [3]. With the development of the artificial
intelligence, the application of deep learning (DL) in RIS-aided systems has
attracted extensive attention. In [6], the authors adopted fully connected
neural networks to estimate the RIS channel and detect the symbols. In [7],
Elbir _et al._ designed a twin convolutional neural network (CNN) to estimate
the direct and the cascaded channel in a RIS-aided communication system.
However, due to the passive characteristics of RIS, the channels from the
source to RIS and that from RIS to destination are coupled, and the size of
the equivalent channel is in scale with the number of the RIS elements, which
is usually large enough to accurately manipulate an incoming electromagnetic
(EM) field. Thus, it would cost many pilot resource to directly achieve the
equivalent channel of large size at destination. Thus, how to reduce the
overhead of the channel estimation is an interesting topic over the RIS-aided
network. Recently, Taha et al. used a small part of RIS elements to sub-sample
the channels, and optimize the beamforming vector of RIS with the channel
estimated at the selected elements [8].

In this paper, we further examine the channel compression over the physical
space for RIS-aided communication. After selecting a fraction of the RIS
elements, we achieve the equivalent channel formed by the source, the
destination and the chosen RIS elements. Then, we extrapolate the channels to
all elements from those estimated at chosen elements, where DL is adopted.
Furthermore, inspired by the ordinary differential equation (ODE), we modify
the structure of the cascaded CNN by adding cross-layer connections, namely
introducing coefficients and linear calculations between the network layers.
The proposed ODE-based CNN can obtain more accurate solutions, and its
performance can be verified to be better than the cascaded CNN.

##  II System And Channel Model

As shown in Fig. 1, let us consider an indoor scenario, where multiple-antenna
base station (BS) communicates with a single-antenna user via RIS reflection.
A BS is equipped with a uniform linear array (ULA) with $M$ antennas. RIS is
in the form of a uniform planar array (UPA) and consists of $L=L_{h}L_{v}$
elements, where $L_{h}$ and $L_{v}$ separately denotes the sizes along the
horizontal and vertical dimensions. Moreover, the orthogonal frequency
division multiplexing (OFDM) scheme is adopted and the number of subcarriers
is $K$. Since the indoor environment is easy to be blocked by objects or
people, the direct channel between the BS and the user may be destroyed. Thus,
we only consider the channel reflected by RIS, i.e., the cascaded channel,
instead of the direct channel. Obviously, the cascaded channel consists of two
parts: the link from the BS to the RIS, i.e.,
$\mathbf{H}\in\mathbb{C}^{L\times M}$, and that from the RIS to the user,
i.e., $\mathbf{g^{\mathrm{H}}}\in\mathbb{C}^{L\times 1}$. The received signal
at the $k$-th subcarrier of the user can be given as

$\displaystyle
y_{k}=\mathbf{g}_{k}\mathbf{\Psi}\mathbf{H}_{k}\mathbf{s}_{k}+n_{k},$ (1)

where $\mathbf{\Psi}\in\mathbb{C}^{L\times L}$ is a diagonal matrix, i.e.,
$\mathbf{\Psi}=\text{diag}\\{\beta_{1}\text{exp}(j\phi_{1}),\dots,\beta_{L}\text{exp}(j\phi_{L})\\}$,
$\mathbf{s}_{k}$ is the downlink $M\times 1$ transmitted signal at the $k$-th
sub-carrier, and $n_{k}\sim\mathcal{CN}(0,\sigma_{n}^{2})$ is the addictive
white Gaussian noise. Due to the lack of signal processing capability at RIS,
$\boldsymbol{\Psi}$ is the same at different sub-carriers. Notice that
$\phi_{i}$ in $\mathbf{\Psi}$ represents the phase shift introduced by each
RIS element while $\beta_{i}$ controls this element’s on-off state, which will
be described in the following. Moreover, the channel between BS and RIS at the
$k$-th subcarrier is given by

$\displaystyle\mathbf{H}_{k}=\frac{1}{\sqrt{K}}\sum_{i=1}^{P_{h}}h_{i,f_{c}}e^{-j2\pi\frac{k\tau_{h,i}}{KT_{s}}}\mathbf{a}_{r}(\phi_{h,i},\theta_{h,i})\mathbf{a}_{t}^{\mathrm{H}}(\psi_{h,i}),$
(2)

where $h_{i,f_{c}}$ is the complex channel gain along the $i$-th scattering
path at the carrier frequency $f_{c}$, $\tau_{h,i}$ is the time delay, and
$\mathbf{a}_{r}(\phi_{h,i},\theta_{h,i})$, $\mathbf{a}_{t}(\psi_{h,i})$ are
the spatial steering vectors, with $\phi_{h,i}$ and $\theta_{h,i}$ as the
azimuth angle and elevation angle of the receiver, respectively, and
$\psi_{h,i}$ as the angle of departure (AoD). Correspondingly,
$\mathbf{a}_{r}(\phi_{h,i},\theta_{h,i})$ can be written as

$\displaystyle\mathbf{a}_{r}(\phi_{h,i},\theta_{h,i})=\mathbf{a}_{el}(\phi_{h,i})\otimes\mathbf{a}_{az}(\phi_{h,i},\theta_{h,i})\in\mathcal{C}^{L\times
1},$ (3)

where the $L_{v}\times 1$ vector
$\mathbf{a}_{el}(\phi_{h,i})=[1,e^{-j2\pi\frac{d}{\lambda_{c}}\cos\phi_{h,i}},\dots,e^{-j2\pi\frac{d}{\lambda_{c}}(L_{v}-1)\cos\phi_{h,i}}]^{\mathrm{T}}$
and the $L_{h}\times 1$ vector
$\mathbf{a}_{az}(\phi_{h,i},\theta_{h,i})=[1,e^{-j2\pi\frac{d}{\lambda_{c}}\sin\phi_{h,i}\cos\theta_{h,i}},\dots,e^{-j2\pi\frac{d}{\lambda_{c}}(L_{h}-1)\sin\phi_{h,i}\cos\theta_{h,i}}]^{\mathrm{T}}$.
$\lambda_{c}$ is the carrier wavelength and $d$ denotes antenna spacing.
Furthermore, $\otimes$ represents the Kronecker product operator and
$[\cdot]^{\mathrm{T}}$ represents the transpose. Moreover,
$\mathbf{a}_{t}(\psi_{h,i})$ can be given by

$\displaystyle\mathbf{a}_{t}(\psi_{h,i})=\frac{1}{\sqrt{M}}[1,e^{j\frac{2\pi}{\lambda_{c}}d\sin\psi_{h,i}},\dots,e^{j\frac{2\pi}{\lambda_{c}}d(M-1)\sin\psi_{h,i}}]^{\mathrm{T}}.$
(4)

Correspondingly, the channel between the RIS and the user at the $k$-th
subcarrier is

$\displaystyle\mathbf{g}_{k}^{\mathrm{H}}=\frac{1}{\sqrt{K}}\sum_{i=1}^{P_{g}}g_{i,f_{c}}e^{-j2\pi\frac{k\tau_{g,i}}{KT_{s}}}\mathbf{a}_{t}^{\mathrm{H}}(\phi_{g,i},\theta_{g,i}),$
(5)

where the structure of $\mathbf{a}_{t}(\phi_{g,i},\theta_{g,i})$ is similar to
that of $\mathbf{a}_{r}(\phi_{h,i},\theta_{h,i})$.

The cascaded channel matrix between BS and the user at the $k$-th subcarrier
can be defined as $\mathbf{C}_{k}=\mathbf{G}_{k}\mathbf{H}_{k}$, where
$\mathbf{G}_{k}=\mathrm{diag}\\{\mathbf{g}_{k}\\}$, and $\mathbf{C}_{k}$ has a
size of $L\times M$. Then, let us define the $LM\times 1$ vector
$\mathbf{c}_{k}=[\mathbf{c}_{k,1}^{\mathrm{T}},\dots,\mathbf{c}_{k,L}^{\mathrm{T}}]^{\mathrm{T}}$,
where $\mathbf{c}_{k,i}$ represents the $i$-th column of $\mathbf{C}_{k}$.
Within the RIS communication system, it is proved that the optimal
$\boldsymbol{\Psi}$ is closely related with all the cascaded channels at $K$
subcarriers, i.e.,
$\mathbf{C}=[\mathbf{c}_{1},\mathbf{c}_{2},\dots,\mathbf{c}_{K}]$ [9]. Hence,
our aim is to estimate $\mathbf{C}$.

Figure 1: RIS-aided communication system in an indoor scene.

##  III Proposed Channel Extrapolation Method

###  III-A Framework Design

Theoretically, we can send a pilot matrix $\mathbf{X}_{k}$ of size $ML\times
S$ to directly recover $\mathbf{c}_{k}$ with the linear estimator, where $S$
represents the time duration of $\mathbf{X}_{k}$. From the Bayesian estimation
theory, we can effectively recover $\mathbf{c}_{k}$ when $S\geq ML$. However,
in massive MIMO systems, both $M$ and $L$ are relatively large. Then, a
significant number of pilot needs to be employed, which drastically decreases
the spectrum efficiency of the transmission. To overcome this bottleneck, we
can utilize a fraction of $N$ RIS elements and sub-sample $\mathbf{C}_{k}$.
Without loss of generality, we set the number of the selected RIS elements as
$N$, which can be implemented through setting the parameters $\beta_{i}$,
$\phi_{i}$ in $\boldsymbol{\Psi}$, $i=1,2,\ldots,L$. Specifically, for the $N$
chosen elements, we set their $\phi_{i}$ and $\beta_{i}$ as $0$ and 1,
respectively. For others, the corresponding amplitude parameter $\beta_{i}$ is
0. After this operation, the size of the sub-sampled cascaded channel at the
$k$-th subcarrier is reduced to $N$. Correspondingly, the sub-sampled cascaded
channel at $K$ subcarriers can be written as
$\widetilde{\mathbf{C}}\in\mathbb{C}^{MN\times K}$. Obviously, compared with
$\mathbf{C}$, a pilot sequence of shorter time duration would be required to
estimate $\widetilde{\mathbf{C}}$.

If the power of the pilot sequence is large enough, we can estimate
$\widetilde{\mathbf{C}}$ with quite high accuracy. However, we should utilize
$\widetilde{\mathbf{C}}$ to infer the unknown cascaded channel at the $L-N$
non-chosen RIS elements. Thus, in the following, we construct a DL-based
framework to extrapolate $\mathbf{C}$ from $\widetilde{\mathbf{C}}$. It may be
worth noting that the selection pattern for the RIS elements can impact the
performance of the channel extrapolation and should be optimized. This topic
is beyond the scope of this paper, though. Further, we adopt the uniform
sampling scheme.

###  III-B ODE-based Channel Extrapolation

As mentioned above, the input of the network is $\widetilde{\mathbf{C}}$, and
its output is $\mathbf{C}$. The task of our network is to learn the mapping
function from $\widetilde{\mathbf{C}}$ to $\mathbf{C}$. In other words, we
want to estimate the complete channel through the sub-sample version, which is
made possible by the correlation between different RIS elements.

The channel extrapolation is similar to the super-resolution in the field of
image processing. For this kind of problem, CNN has great advantages and is
very suitable to use the correlation between data elements for information
completion. In order to get better network performance, we can increase the
number of data layers or modify the network structure. However, more layers
will result in higher calculational requirements. Moreover, when the number of
layers reaches a certain number, the improvement become less and less.
Sometimes, the excessive deepening of the network causes the gradient
explosion and disappearance. Thus, optimizing the network structure is more
widely used than simply deepening the neural network. Theoretically, if we add
some proper connections between layers, the performance of the network may be
better, like residual neural network (ResNet) [10].

Recently, ODE have been introduced to the neural network and utilized to
describe the latent relation between different data layers [11]. With such
powerful characterization, we could speed up the convergence and learning
performance of the CNN. Moreover, with the development of mathematical
science, it is possible to use the numerical solutions of differential
equations to modify the network structure and obtain possible gains. Here, we
incorporate two numerical approximation methods, i.e., LeapFrog and Runge-
Kutta methods, into CNN. The main difference between them lies in the
approximation accuracy.

$\mathbf{LeapFrog\ method}$ : LeapFrog method is a second-order approximation
scheme and can be written as

$\displaystyle y_{n+1}=y_{n-1}+2hf(x_{n},y_{n}),$ (6)

where $f(x_{n},y_{n})$ denotes the derivative at $(x_{n},y_{n})$ and $2h$ can
be seen as an interval of width $x_{n+1}-x_{n-1}$. Applying (6) for the CNN,
we can connect the ($n+1$)-th layer with the ($n-1$)-th one. The corresponding
relationship can be formulated as:

$\displaystyle\mathbf{D}_{n}=\mathbf{D}_{n-2}+G(\mathbf{D}_{n-1}),\
n=3,4,5,\dots,$ (7)

where $\mathbf{D}_{i}$ represents the output data of the $i$-th layer, and
$G(\cdot)$ is an operation containing a ReLu activation function, a
convolution layer and a multiplier.

###### Remark 1

The LeapFrog method is an improved version of the forward Euler equation,
which can be written as $y_{n+1}=y_{n}+hf(x_{n},y_{n})$. The forward Euler
method is the simplest first-order approximation of ODE and has a similar
structure with ResNet.

$\mathbf{Runge-Kutta\ methods}$ : The theory of the numerical ODEs suggests
that a higher-order approximation results in less truncation error and higher
accuracy. Hence, we turn to the Runge-Kutta methods, which are common
numerical methods for ODE and can be expressed as [12]

$\displaystyle y_{n+1}=y_{n}+\sum_{i=1}^{I}\gamma_{i}G_{i},$ (8)

and $G_{1}=hf(x_{n},y_{n})$, while $G_{i}$ has the form of

$\displaystyle
G_{i}=hf(x_{n}+\alpha_{i}h,y_{n}+\sum_{j=1}^{i-1}\beta_{ij}G_{j}),\
i=2,3,\cdots,n,$ (9)

where $I$ represents the number of stages; $\alpha_{i}$ and $\beta_{i}$, like
$\gamma_{i}$ in (8), are the related coefficients of the $i$-th stage.

If we set
$\gamma_{1}=\frac{1}{6},\gamma_{2}=\frac{2}{3},\gamma_{3}=\frac{1}{6},\alpha_{2}=\frac{1}{2},\beta_{21}=\frac{1}{2},\alpha_{3}=1,\beta_{31}=-1,\beta_{32}=2$,
we obtain the 3-stage Runge-Kutta equation as the basic structure in our work,
which can be expressed as

$\displaystyle y_{n+1}$ $\displaystyle=y_{n}+\frac{1}{6}(G_{1}+4G_{2}+G_{3}),$
(10)

where $G_{1}$, $G_{2}$, $G_{3}$ can be separately written as

$\displaystyle G_{1}$ $\displaystyle=hf(x_{n},y_{n}),$ (11) $\displaystyle
G_{2}$ $\displaystyle=hf(x_{n}+\frac{h}{2},y_{n}+\frac{1}{2}G_{1}),$ (12)
$\displaystyle G_{3}$ $\displaystyle=hf(x_{n}+h,y_{n}-G_{1}+2G_{2}).$ (13)

Figure 2: (a) Our proposed network architecture; (b) The structure of the
RK3-Block; (c) The architecture of operation $G$, which contains two ReLu
activation functions and two convolutional layers.

With (10) - (13), we can construct an improved CNN structure, referred to as
RK3-Block and depicted in Fig. 2 (b). Correspondingly, the constraints among
different layers in this block can be written as

$\displaystyle\mathbf{D}_{1}=$ $\displaystyle\
\mathbf{D}_{0}+\frac{1}{2}G^{\prime}(\mathbf{D}_{0}),\
\mathbf{D}_{2}=\mathbf{D}_{0}-G^{\prime}(\mathbf{D}_{0})+2G^{\prime}(\mathbf{D}_{1}),$
(14)
$\displaystyle\mathbf{D}_{3}=\mathbf{D}_{0}+\frac{1}{6}(G^{\prime}(\mathbf{D}_{0})+4G^{\prime}(\mathbf{D}_{1})+G^{\prime}(\mathbf{D}_{2})),$
(15)

where the positions of $\mathbf{D}_{i}$, $i=0,1,2,3$, are presented in Fig. 2
(b), and the operation $G^{\prime}(\cdot)$ contains two ReLu activation
functions and two convolution layers. Similar to RK3-Block, with (7), we can
obtain a modified CNN structure from the LeapFrog approximation and refer to
it as LF-Block, which includes three data layers. As shown in Fig. 2 (a), we
can cascade several RK3-Blocks or LF-Blocks to deepen the network for better
results.

###  III-C Learning Scheme

The valid input of our network is the sub-sampled channel
$\widetilde{\mathbf{C}}\in\mathbb{C}^{MN\times K}$, and the label is the
entire cascaded channel $\mathbf{C}\in\mathbb{C}^{ML\times K}$. In order to
facilitate the training and the generation of data, we set the entries of
$\mathbf{C}$ related with the $L-N$ non-chosen RIS elements as 0 and obtain
the resultant matrix $\mathbf{C}^{o}$, whose non-zero entries are same with
those of $\widetilde{\mathbf{C}}$. Correspondingly, we treat $\mathbf{C}^{o}$
as the raw input of the ODE-based CNN.

Then, we reshape the raw input data and the label of the network as
$\mathbf{Z}_{\mathrm{IN}}=[\Re({\mathbf{C}^{o}});\Im({\mathbf{C}^{o}})]$ and
$\mathbf{Z}_{\mathrm{TA}}=[\Re(\mathbf{C});\Im(\mathbf{C})]$, respectively.
Both $\mathbf{Z}_{\mathrm{IN}}$ and $\mathbf{Z}_{\mathrm{TA}}$ are real-valued
matrices with the size of $ML\times K\times 2$. Correspondingly, the output of
this network can be written as
$\mathbf{Z}_{\mathrm{OUT}}=[\Re(\widehat{\mathbf{C}});\Im(\widehat{\mathbf{C}})]\in\mathbb{C}^{ML\times
K\times 2}$, where $\widehat{\mathbf{C}}$ is the estimate of ${\mathbf{C}}$.
In our proposed network, there are $N_{c}$ convolutional layers. In the $n$-th
layer, the input is processed by $N_{k}$ convolutional kernels of size
$H\times W$. Note that $H$ and $W$ represent the height and the width of the
convolutional kernels. Normally, the size of the output data in each
convolutional layer depends on $H$ and $W$, and it is usually slightly smaller
than the input data.

During the learning stage, the parameter vector
$\boldsymbol{\omega}=[\boldsymbol{\omega}_{1}^{T},\boldsymbol{\omega}_{2}^{\mathrm{T}},\dots,\boldsymbol{\omega}_{N_{c}}^{T}]^{T}$
is optimized by minimizing the mean squared error (MSE) between the output
$\mathbf{Z}_{\mathrm{OUT}}$ and the target $\mathbf{Z}_{\mathrm{TA}}$, where
the vector $\boldsymbol{\omega}_{n}$ contains all the model parameters of the
$n$-th layer, $n=1,2,\ldots,N_{c}$. Hence, the loss function can be written as

$\displaystyle\mathcal{L}=\frac{1}{M_{b}MLK}\sum_{i=1}^{M_{b}}\begin{Vmatrix}[\mathbf{Z}_{\mathrm{TA}}]_{i}-[\mathbf{Z}_{\mathrm{OUT}}]_{i}\end{Vmatrix}_{F}^{2},$
(16)

where $||\mathbf{A}||_{F}$ is the $\mathrm{Frobenius}$ norm of matrix
$\mathbf{A}$ and $M_{b}$ denotes the batch size for training. Here, the
adaptive moment estimation (Adam) [13] algorithm is adopted to achieve the
best $\boldsymbol{\omega}$, which is controlled by the learning rate $\eta$.

TABLE I: Layer Parameters for the CNN with ODE-RK3 Structure.

Layer | Output size | Activation | Kernel size | Strides
---|---|---|---|---
$1\times$ Conv2D | $256\times 64\times 128$ | None | $5\times 5$ | $1\times 1$
$4\times$ RK3-Block | $256\times 64\times 128$ | ReLu | $3\times 3$ | $1\times 1$
$1\times$ Conv2D | $256\times 64\times 2$ | None | $3\times 3$ | $1\times 1$

TABLE II: Performance Comparison of Different Structures under Different
Sampling Rates.

Sampling Rate | Method
---|---
| ODE-RK3
---
(Loss / NMSE)
| ODE-LF
---
(Loss / NMSE)
| CNN
---
(Loss / NMSE)
1/2 | 0.00001 / -39.59dB | 0.00002 / -35.53dB | 0.00003 / -32.37dB
1/4 | 0.00002 / -33.77dB | 0.00009 / -28.18dB | 0.00012 / -26.78dB
1/8 | 0.00086 / -18.12dB | 0.00133 / -16.25dB | 0.00151 / -15.69dB
1/16 | 0.0155 / -5.7dB | 0.01834 / -4.95dB | 0.01929 / -4.69dB

##  IV Simulation Results

Figure 3: The NMSE of channel extrapolation versus epochs.

In this section, we evaluate the channel extrapolation performance of ODE-
based CNN through numerical simulation. We first describe the communication
scenario and dataset source, and then show the parameters of the training
network. Finally, we show the simulation results and explain the performance
of our proposed network.

The scenario we consider is an indoor scene with user, BS and RIS. To generate
this scenario, we resort to the indoor distributed massive MIMO scenario I1 of
the DeepMIMO dataset, which is generated based on the Wireless InSite [14].

The ULA at BS has $4$ antennas, i.e., $M=4$, while the size of the RIS’s UPA
is $8\times 8$, i.e., $L=64$. The carrier frequency of channel estimation is
$2.4$ GHz. The OFDM signal bandwidth is set as $20$ MHz, while the number of
subcarriers is $K=64$. The antenna spacing is $\frac{\lambda}{2}$, and the
number of paths is $5$. Furthermore, the activated users are located from the
$1$-st row to the $100$-th row. Each row contains $201$ users, and the total
number of users is $20100$. The users are split in two parts, i.e., the
training and the test groups, according to the ratio $4:1$. The sampling rate
$r=\frac{N}{L}$ is separately set as $\frac{1}{2},\frac{1}{4},\frac{1}{8}$ and
$\frac{1}{16}$.

In the simulations, we adopt three network structures for comparison, i.e.,
the ODE-RK3 structure formed by some RK3-Blocks, the ODE-LF structure
containing several LF-Blocks and the cascaded CNN network. For fairness, all
CNNs have $26$ layers and the same number of parameters. The ODE-based network
contains $4$ RK3-Blocks or LF-Blocks (each block consists of $6$ convolutional
layers), a head convolutional layer and a tail convolutional layer.
Considering ODE-RK3 structure as an example, we list the layer parameters of
the CNN in TABLE I. Specially, in the hidden layers, the number of neurons is
$128$, and $\mathrm{ReLU}$ is adopted as the activation function, i.e.,
$\mathrm{ReLU}(x)=\max(x,0)$. The kernel size of the first convolutional layer
is $5\times 5$, and that of the remainder convolutional layers is set as
$3\times 3$. The learning rate $\eta$ is initialized as $0.0005$ and decreases
with increased iteration times. Specifically, after $40$ iterations, the
learning rate reduces by $20\%$ for every $10$ epochs.

TABLE II shows the performance of different network structures. It can be
noted that the proposed ODE CNN is always superior to the cascaded CNN
network, and this gain enhances with the increase of the sampling rate $r$.
The RK3 structure with three-order performs better than the third-order LF
structure. When sampling rates are $\frac{1}{2}$ and $\frac{1}{4}$, the ODE-
RK3 structure can achieve satisfactory results. Furthermore, in terms of
channel extrapolation normalized MSE (NMSE), the ODE-RK3 network at
$r=\frac{1}{4}$ performs better than the CNN network at $r=\frac{1}{2}$, which
means that the length of the pilot for the sub-sampled channel
$\widetilde{\mathbf{C}}$ estimation can be reduced through introducing the ODE
structure. If the compression ratio is relatively low, such as
$r=\frac{1}{16}$, the performance of the ODE-based CNN is not significantly
better than that of the cascaded CNN due to the reduced raw input information
for the channel extrapolation.

Figure 4: The loss of channel extrapolation versus epochs.

Fig. 3 depicts the curves of NMSE with respect to the number of epochs. Two
structures (the ODE-RK3 and cascaded CNN structures) and two sample rates
($\frac{1}{4}$ and $\frac{1}{8}$) are considered here. It can be seen that
with the increase of iteration time, all the NMSE curves present a downward
trend and reach the stable levels after $115$ epochs. Furthermore, for a given
sampling rate, the NMSEs of the ODE-based CNN are always lower than those of
the cascaded CNN. Fig. 4 depicts the training loss of different CNN structures
versus epochs. It can be checked from Fig. 4 that, with fixed rate $r$, the
training loss in the ODE-RK3 network decreases faster than that in the
cascaded CNN network, which means that the ODE-based network can be trained
more quickly than CNN.

Figure 5: The NMSEs of the channel extrapolation verus the frequency gaps.

So far, we considered the channel extrapolation at the same frequency band.
However, our proposed scheme can still be used for the case with frequency
difference. In actual systems, such as the frequency division duplexing
system, the uplink and downlink channels operate in different frequency bands.
Fig. 5 shows the performance of the ODE-RK3 and the cascaded CNN structures
under different frequency gaps. As can be seen from Fig. 5, both ODE-RK3 and
CNN structures are affected by the frequency gaps. As the frequency difference
increases, the NMSE of channel extrapolation slightly augments. It is worth
noting that the ODE-RK3 structure always performs better than cascaded CNN,
which proves the stability and effectiveness of the proposed ODE-based CNN.

##  V Conclusion

In this paper, we have examined a RIS-assisted MIMO communication system, and
designed an ODE-based CNN to extrapolate the cascaded channel. In our scheme,
only part of the full CSI is needed. Hence, some of the RIS elements could be
turned off through spatial sampling, which greatly reduces the length of the
pilot sequence in the channel estimation phase and improves the resource
utilization. Simulation results have demonstrated that the proposed
extrapolation scheme can effectively compress the large-scale RIS channel over
the physical space. Moreover, the ODE-based structure can speed up the
convergence and improve the performance of the cascaded CNN.

## References

  * [1] F. Rusek, D. Persson, B. K. Lau, E. G. Larsson, T. L. Marzetta, O. Edfors, and F. Tufvesson, “Scaling up MIMO: Opportunities and challenges with very large arrays,” _IEEE Signal Process. Mag._ , vol. 30, no. 1, pp. 40–60, Jan. 2013.
  * [2] W. Yan, X. Yuan, Z. He, and X. Kuai, “Passive beamforming and information transfer design for reconfigurable intelligent surfaces aided multiuser MIMO systems,” _IEEE J. Sel. Areas Commun._ , pp. 1–1, 2020.
  * [3] J. Ma, S. Zhang, H. Li, F. Gao, and S. Jin, “Sparse Bayesian learning for the time-varying massive MIMO channels: Acquisition and tracking,” _IEEE Trans. Commun._ , vol. 67, no. 3, pp. 1925–1938, Mar. 2019.
  * [4] K. Ardah, S. Gherekhloo, A. L. F. de Almeida, and M. Haardt, “TRICE: An efficient channel estimation framework for RIS-aided MIMO communications,” arXiv:2008.09499, 2020. [Online]. Available: https://arxiv.org/abs/2008.09499.
  * [5] J. He, H. Wymeersch, and M. Juntti, “Channel estimation for RIS-aided mmWave MIMO systems via atomic norm minimization,” arXiv:2007.08158, 2020. [Online]. Available: https://arxiv.org/abs/2007.08158.
  * [6] S. Khan, K. S Khan, N. Haider, and S. Y. Shin, “Deep-learning-aided detection for reconfigurable intelligent surfaces,” arXiv:1910.09136, 2020. [Online]. Available: https://arxiv.org/abs/1910.09136.
  * [7] A. M. Elbir, A. Papazafeiropoulos, P. Kourtessis and S. Chatzinotas, “Deep channel learning for large intelligent surfaces aided mm-wave massive MIMO systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 9, pp. 1447-1451, Sept. 2020.
  * [8] A. Taha, M. Alrabeiah, and A. Alkhateeb, “Enabling large intelligent surfaces with compressive sensing and deep learning,” arXiv:1904.10136v2, 2019. [Online]. Available: https://arxiv.org/abs/1904.10136v2.
  * [9] S. Lin, B. Zheng, G. C. Alexandropoulos, M. Wen, M. Di Renzo, and F. Chen, “Reconfigurable intelligent surfaces with reflection pattern modulation: Beamforming design and performance analysis,” _IEEE Wireless Commun. Lett._ , pp. 1-1, Oct. 2020.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition,_ Las Vegas, NV, pp. 770-778, Dec. 2016.
  * [11] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, “Neural ordinary differential equations,” arXiv:1806.07366v5, 2019. [Online]. Available: https://arxiv.org/abs/1806.07366.
  * [12] X. He, Z. Mo, P. Wang, Y. Liu, M. Yang, and J. Cheng, “ODE-inspired network design for single image super-resolution,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , Long Beach, CA, pp. 1732-1741, Jun. 2019.
  * [13] O. P. Kingma and J. Ba, “ADAM: A method for stochastic optimization,” arXiv:1412.6980, 2014, [Online]. Available: https://arxiv.org/abs/1412.6980
  * [14] A. Alkhateeb, “DeepMIMO: A generic deep learning dataset for millimeter wave and massive MIMO applications,” in Proc. Information Theory and Applications Workshop (ITA), San Diego, CA, pp. 1-8, Feb. 2019.


# PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion

Ying-Tian Liu1 Guan Luo1 Heyi Sun1 Wei Yin3 Yuan-Chen Guo1,2111Internship at
VAST. Song-Hai Zhang1222Corresponding author.
1 BNRist, Tsinghua University 2 VAST 3 The University of Adelaide

###### Abstract

Diffusion models trained on large-scale text-image datasets have demonstrated
a strong capability of controllable high-quality image generation from
arbitrary text prompts. However, the generation quality and generalization
ability of 3D diffusion models is hindered by the scarcity of high-quality and
large-scale 3D datasets. In this paper, we present PI3D, a framework that
fully leverages the pre-trained text-to-image diffusion models’ ability to
generate high-quality 3D shapes from text prompts in minutes. The core idea is
to connect the 2D and 3D domains by representing a 3D shape as a set of Pseudo
RGB Images. We fine-tune an existing text-to-image diffusion model to produce
such pseudo-images using a small number of text-3D pairs. Surprisingly, we
find that it can already generate meaningful and consistent 3D shapes given
complex text descriptions. We further take the generated shapes as the
starting point for a lightweight iterative refinement using score distillation
sampling to achieve high-quality generation under a low budget. PI3D generates
a single 3D shape from text in only 3 minutes and the quality is validated to
outperform existing 3D generative models by a large margin.

Figure 1: Our method, PI3D, is able to generate the pseudo-images of a 3D
shape in seconds from text prompts and further refine it in 3 minutes to
achieve better quality.

##  1 Introduction

With recent advances in generative modeling, various models have emerged
capable of capturing complex data distributions across different modalities
such as audio, video, and images. Notably, pioneering image generation models
like Imagen [32], DALL-E [29, 30], Stable Diffusion [31], and others
demonstrate the ability to generate highly realistic and natural-looking
images from text prompts, often indistinguishable from those produced by
humans.

While algorithmic progress plays a crucial role in these achievements, it’s
essential to acknowledge the foundational role of large-scale datasets [34] in
facilitating such applications. Unlike 2D images, 3D objects face inherent
challenges in data acquisition. The labor-intensive and time-consuming
processes involved in 3D scanning and creation make gathering 3D data notably
challenging. Although datasets like Objaverse [9] and Objaverse-XL [8] have
recently been released with millions of 3D assets, they suffer from
insufficient filtering and processing, leading to the inclusion of duplicate,
low-quality, and texture-less content. The absence of a high-quality,
abundant, and semantically rich 3D dataset remains a significant obstacle in
directly constructing 3D generative models.

In order to generate 3D assets in scenarios where 3D priors are difficult to
obtain, researchers have recently been keen to utilize various 3D
representations [24, 41, 21, 35] to lift 2D diffusion models with Score
Distillation Sampling (SDS) [27, 40]. However, applying only 2D prior to
optimization is prone to hinder the 3D nature, inevitably compromising 3D
fidelity and consistency. 3D knowledge is a key issue that cannot be bypassed
for high-quality 3D generation.

The pivotal question arises: can a 3D diffusion model exhibit robust
generalization when 3D data cannot reach the scale and quality of image data?
We answer this question by presenting PI3D, a framework that fully utilizes
the knowledge in pre-trained text-to-image diffusion models and image datasets
to generate high-quality 3D shapes conditioned on text prompts.

\begin{overpic}[scale={0.4}]{figure/triplane-img.png}
\put(0.0,-15.0){Orthogonal Views} \put(110.0,-15.0){Pseudo-Images}
\end{overpic}

Figure 2: We show the 3 orthogonal rendering results and the triplane
representation as 6 pseudo-images for the same scene. We can observe semantic
congruence between them, such as the contours of different parts.

Given the observation that the triplane representation for 3D scenes [26, 4]
shares semantic congruence with orthogonal rendered images, we believe that
the architecture of a pre-trained image diffusion model is also capable of
generating high-quality triplanes. In order to borrow knowledge from 2D
generation, we treat every 3 channels of the triplane representation as one
image to form a set of pseudo RGB images (see Fig. 2). To build our 3D
diffusion model, we start from a pre-trained text-to-image diffusion model and
keep most of its architecture to inherit the generalizability while changing
the self-attention layer to global attention performed on all pseudo-images.
We inject 3D knowledge into the pre-trained 2D diffusion model by fine-tuning
it on the pseudo-images obtained from 20k text-3D pairs from the Objaverse
dataset [9]. Further, we find that the simultaneous use of 2D image data in
training significantly improves generalization ability on complex concepts
while keeping the requirements of the magnitude of 3D data at a low level.
Based on the pseudo-image diffusion model, we can create brand-new 3D objects
from text prompts by performing diffusion sampling in seconds. We take the
sampled object as the starting point for a subsequent refinement process where
score distillation sampling (SDS) [27] is used to further improve the quality.
The refinement process converges quickly and maintains good 3D consistency
thanks to the strong initialization provided by the pseudo-image diffusion
model. PI3D is able to generate high-quality 3D objects from text prompts
within 3 minutes, bringing new possibilities for 3D content creation.

The main contributions of this paper are as follows:

  * •

We propose PI3D, a framework that fully leverages prior knowledge from 2D
diffusion models for high-quality and text-aligned 3D generation in minutes.

  * •

We propose to represent a 3D shape as a set of pseudo-images and adapt a 2D
diffusion model to directly output such a representation, enabling fast
sampling of 3D objects from texts while providing strong initialization for
further refinement.

  * •

PI3D is demonstrated to be effective in 3D generation with respect to visual
quality, 3D consistency, and generation speed.

##  2 Related Work

In this section, we will primarily review recent text-to-3D research. They can
be broadly categorized based on the data they utilize and how they incorporate
3D knowledge: methods using 3D diffusion methods and lifting 2D diffusion
models.

###  2.1 Diffusion-based 3D Generative Model

This line of research focuses on training 3D diffusion models on popular 3D
datasets [5, 9, 8], each with distinct representations of 3D objects. As these
models have access to 3D raw data, they inherently acquire 3D knowledge
inductively from the dataset. Point·E [25] trains two point-cloud diffusion
models conditioned on the text prompt or single-view image to generate a low-
resolution point cloud and up-sample it respectively. It can further obtain
meshes from the point cloud via SDF prediction. Shap·E [19] trains the
diffusion model on the 3D implicit function. It is capable of simultaneously
producing NeRF and SDF with two heads, but it struggles with semantics when
faced with complex prompts. NFD [37], 3DGen [11], and DiffTF [3] adopt
diffusion on the triplane representation to achieve highly efficient mesh
generation but they constrain the generation to be category-specific on
ShapeNet [5] and OmniObject3D [43]. All of these models actually consider only
3D data, which leads to their poor ability to generate reasonable objects when
confronted with some complex concepts.

###  2.2 3D Knowledge in Lifting 2D Diffusion Models

This line of work is typically focused on optimizing a 3D representation based
on a pre-trained 2D diffusion model. The foundational technique involved is
Score Distillation Sampling (SDS), first proposed in [27, 40]. SDS aims to
align the rendering results of a parameterized renderer with the distribution
described by a pre-trained image diffusion model, achieving parameter space
sampling. In DreamFusion [27] and SJC [40], 3D knowledge originates solely
from spatial representations and the view-dependency prompting, which are
relatively weak. This limitation increases the likelihood of 3D inconsistency,
such as the Janus problem, since there is no direct informative constraint
between different views.

Perp-Neg [2] mitigates this problem by correcting the direction of parameter
updates guided by the 2D diffusion model but leaves the lack of a 3D prior
unsolved. Zero-1-to-3 [22] proposes to train a 2D viewpoint transformation
model conditioned on the input view and the relative camera transformation,
injecting implicit 3D knowledge into the 2D diffusion model. The viewpoint-
conditioned diffusion model can be further used for generating 3D consistent
objects from a single view via SDS. Magic123 [28] fuses and balances guidance
from pre-trained 2D diffusion model [31] and viewpoint transformation model
[22], achieving detailed image-to-3D results. MVDream [36] models the
distribution of multi-view images and thus greatly improves the stability of
existing 2D-lifting methods. Although methods in this section are usually
capable of generating an attractive appearance, they all require tons of
optimizing iterations and are of lower diversity.

##  3 Preliminary

Diffusion models [38, 39, 15] are a class of generative models capable of
generating diverse samples from pure noise by progressively denoising it.
Given a sample $z_{0}$ from the real data distribution, a forward diffusion
process is defined as gradually adding noise to $z_{0}$ to form a sequence
$\\{z_{t}=\alpha_{t}z_{0}+\sigma_{t}\epsilon|t=1,2,...,T\\}$, where
$\epsilon\sim\mathcal{N}(0,1)$ is a gaussian noise. Here,
$\alpha_{t},\sigma_{t}$ are pre-defined coefficients ensuring that $z_{t}$ is
close to the real sample at the start and becomes Gaussian noise towards the
end of the forward process, respectively. To generate samples, a diffusion
model $\epsilon_{\phi}(\cdot;t)$ is trained to approximate the reverse process
and gradually denoise the sample by predicting the noise level at the timestep
$t$. A common training objective is a weighted error prediction MSE loss:

$\mathcal{L}=\mathbb{E}_{t\sim[1,...,T],\epsilon\sim\mathcal{N}(0,1)}[w(t)||\epsilon_{\phi}(\alpha_{t}z_{0}+\sigma_{t}\epsilon;t)-\epsilon||_{2}]$
(1)

##  4 Method

In this section, we mainly focus on how PI3D is built. It consists of two
parts as a whole, the first part is a pseudo-image diffusion model that is
responsible for fast sampling of 3D objects. It follows two stages similar to
many other latent diffusion methods [11, 18, 7, 19]. In the first stage (in
Sec. 4.1), a 3D representation for each 3D object is fitted and next (in Sec.
4.2), we build a text-conditioned pseudo-image diffusion model adapting from a
pre-trained 2D diffusion model. The second part (in Sec. 4.3) is a lightweight
refinement using Score Distillation Sampling (SDS) that converges quickly and
further improves the quality of the sampled 3D object.

\begin{overpic}[scale={0.4}]{figure/triplane.pdf} \put(120.0,100.0){$r$}
\put(82.0,80.0){\small$p$} \put(85.0,-10.0){$\mathcal{L}_{\text{hull}}$}
\put(150.0,-10.0){$\mathcal{L}_{\text{color}}$}
\put(185.0,-10.0){$\mathcal{L}_{\text{mask}}$}
\put(215.0,-10.0){$\mathcal{L}_{\text{depth}}$} \end{overpic} Figure 3: The
illustration of our depth-aware triplane fitting.

###  4.1 Depth-Aware Triplane Fitting

We represent each object as three axis-aligned feature maps
$F_{xy},F_{xz},F_{yz}\in\mathbb{R}^{C\times H\times W}$. Specifically, we set
$C=6$ to form 6 pseudo-images. The spatial feature $F(p)$ for any query point
$p$ is constructed by concatenating the features bi-linearly interpolated on
the three maps at the projected 2D positions.

$F(p)=[F_{xy}(p_{xy}),F_{xz}(p_{xz}),F_{yz}(p_{yz})]$ (2)

The color and density at the point $p$ are then decoded by a shared MLP
$\mathcal{D}(\cdot;\theta)$.

$c(p),\sigma(p)=\mathcal{D}(F(p);\theta)$ (3)

The expected color of a camera ray $r$ is calculated in a volume rendering
manner [20, 24].

$\displaystyle\hat{C}(r)=$
$\displaystyle\sum_{i=1}^{N}T_{i}(1-\exp(-\delta\sigma(p_{i})))c(p_{i}),$ (4)
$\displaystyle\text{where}\ T_{i}=\exp(-\sum_{j=1}^{i-1}\sigma\delta(p_{j}))$

$\\{p_{i}\\}_{i=1}^{N}$ denotes the sampled points along that ray. We do not
involve additional positional embedding like in vanilla NeRF [24], since we
want the decoder could be position-agnostic, ensuring that what is similar on
the rendered images is also similar on the triplane. To encourage a correct
underlying geometry, we also supervise the object mask $\hat{M}(r)$ and the
expected depth $\hat{D}(r)$ along that ray, which are given by:

$\displaystyle\hat{M}(r)$
$\displaystyle=\sum_{i=1}^{N}T_{i}(1-\exp(-\delta\sigma(p_{i})))$ (5)
$\displaystyle\hat{D}(r)$
$\displaystyle=\sum_{i=1}^{N}T_{i}(1-\exp(-\delta\sigma(p_{i})))t(p_{i})$

where $t(p)$ is the distance from the ray origin to $p$.

The overall loss on rendering views is as follows:

$\displaystyle\mathcal{L}_{\text{color}}$
$\displaystyle=\sum_{r}(||C(r)-\hat{C}(r)||_{2}+\lambda_{\text{L1}}||C(r)-\hat{C}(r)||_{1})$
(6) $\displaystyle\mathcal{L}_{\text{mask}}$
$\displaystyle=\lambda_{\text{mask}}\sum_{r}\text{BCE}(\hat{M}(r),M(r))$
$\displaystyle\mathcal{L_{\text{depth}}}$
$\displaystyle=\lambda_{\text{depth}}\sum_{r}(M(r)\otimes||\hat{D}(r)-D(r)||_{1})$

where $\otimes$ is element-wise production. We use both MSE and L1 loss on the
rendered color for better convergence.

In addition to encouraging the triplane to fit the object well, we still want
it to be friendly to the later diffusion process. So it is important to put
several regularizations on the triplane itself. Inspired by [3], we use L2 and
TV loss as follows:

$\displaystyle\mathcal{L}_{\text{L2}}=$ $\displaystyle||F||_{2}$ (7)
$\displaystyle\mathcal{L}_{\text{TV}}=$ $\displaystyle\sum_{1\leq
i<H}||F(i,\cdot)-F(i+1,\cdot)||_{1}$ $\displaystyle+$
$\displaystyle\sum_{1\leq i<W}||F(\cdot,i)-F(\cdot,i+1)||_{1}$

In order for the geometry to converge fast, we render 3 axis-aligned views in
orthogonal projection and adopt these masks $O_{xy},O_{yz},O_{xz}$ on 3 planes
to constraint where it has non-zero features, which is basically equivalent to
a 3-view visual hull constraint.

$\mathcal{L}_{\text{hull}}=||(1-O)\otimes F||_{1}$ (8)

Above we neglect the subscript $xy,xz,yz$ as they are adopted on all planes.

In general, we adopt an autodecoder architecture to fit all the triplanes.
However, a large number of triplanes cannot be put into the GPU memory at the
same time. So we first train the shared MLP $\mathcal{D}(\cdot;\theta)$ on a
subset and fit all the 3D assets independently by freezing
$\mathcal{D}(\cdot;\theta)$. The final optimization objective for either
training $\mathcal{D}$ or $F$ is

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{color}}$
$\displaystyle+\lambda_{\text{mask}}\mathcal{L}_{\text{mask}}+\lambda_{\text{depth}}\mathcal{L}_{\text{depth}}$
(9)
$\displaystyle+\lambda_{\text{L2}}\mathcal{L}_{\text{L2}}+\lambda_{\text{TV}}\mathcal{L}_{\text{TV}}+\lambda_{\text{hull}}\mathcal{L}_{\text{hull}}$

\begin{overpic}[scale={0.4}]{figure/large.pdf} \put(40.0,810.0){\Large
Point\textperiodcentered E~{}\cite[cite]{[\@@bibref{Number}{pointe}{}{}]}}
\put(150.0,810.0){\Large Shap\textperiodcentered
E~{}\cite[cite]{[\@@bibref{Number}{shape}{}{}]}}
\put(270.0,810.0){\Large{Ours}} \par\put(370.0,810.0){\Large
Point\textperiodcentered E~{}\cite[cite]{[\@@bibref{Number}{pointe}{}{}]}}
\put(490.0,810.0){\Large Shap\textperiodcentered
E~{}\cite[cite]{[\@@bibref{Number}{shape}{}{}]}}
\put(600.0,810.0){\Large{Ours}} \end{overpic}

Figure 4: Examples of 3D models generated by Point·E [25], Shap·E [19] and our method conditioned on various text prompts. 3D models are presented by 3 rendered views. We follow Point·E [25] to extract the surface for the point cloud rendering. The prompts are listed below each sample.  Method | ViT-B-32 | ViT-L-14
---|---|---
CLIP Score | R-Precision@1 | R-Precision@10 | CLIP Score | R-Precision@1 | R-Precision@10
Point·E | 22.3 | 4.6 | 15.7 | 42.6 | 5.7 | 26.3
Shap·E | 62.0 | 16.0 | 41.0 | 48.3 | 17.3 | 43.7
PI3D | 65.9 | 25.2 | 56.7 | 53.8 | 29.3 | 63.3
Table 1: Quantitative comparisons between PI3D and state-of-the-art text-to-3D
generative models on CLIP Score [13] and CLIP R-Precision metrics. The
R-precision is calculated based on top 1 and top 10 recall.

###  4.2 Pseudo-Image Diffusion

We choose Stable Diffusion [31] as our base model and build a pseudo-image
diffusion model on top of it without altering its architecture. The two main
challenges here are: (1) how to simultaneously output multiple multi-channel
feature maps under the architecture, and (2) how to model the cross-plane
consistency to form a reasonable underlying 3D geometry. We view each triplane
representation as a set of 6 pseudo-images and stack them along the batch
dimension as the model input. A global pseudo-image attention block, inspired
by video and multi-view diffusion models [16, 42, 17, 36], is adapted from the
original self-attention block without changing the parameters to aggregate
tokens on 6 pseudo-images. In this way, we enrich the feature interaction
across three planes and incorporate the 3D knowledge into the global attention
block. So far, all the attention layers are order-invariant. Thus, to identify
the input images, we allocate a learnable embedding for each pseudo-image and
add it to the timestep embedding in the cross-attention block.

We directly trained the model on our fitted triplanes but observed a decline
in generalization ability on 3D assets over prolonged training, particularly
for prompts involving combined concepts and multiple objects, showing less
robustness than image diffusion models. We believe the performance degradation
is attributed to the quality and scale of 3D data. There are basically all
single objects in the Objaverse dataset [9] and still exist many misaligned
text descriptions in Cap3D [23]. To enhance robustness, we train the pseudo-
image diffusion with additional real images from LAION-5B [34] by disabling
global attention blocks and learnable embedding. We observe obvious
improvements in the diversity and prompt alignment after training with mixed
2D-3D data which we will discuss later in the ablation study (see Sec. 5.3).

###  4.3 Lightweight Refinement

Upon obtaining the diffusion model, two intuitive ways to generate 3D models
from input texts are considered. One can directly sample new 3D models within
seconds from it and further refine it with 2D SDS. Alternatively, we can view
the 3D diffusion model as a prior and jointly optimize a triplane-based NeRF
with 2D and 3D SDS, like Magic123 [28] to guarantee the 3D consistency. In
practice, we find that sampling a 3D model first using our method and then
refining it is a simple yet effective way to produce high-quality objects and
guarantee 3D consistency. We continue to optimize the triplane generated by
the pseudo-image diffusion along with the shared decoder with SDS loss. To
prevent the 3D shape from being over-optimized, we refine the appearance and
geometry by sampling only small noise levels in SDS. Thanks to a good
initialization, it significantly accelerates the convergence compared to
previous optimization-based methods [27, 6] and thus arrives at efficient 3D
content generation.

##  5 Experiments

###  5.1 Implementation Details

#### Data

For the 3D dataset, we use 20k 3D objects from Objaverse [9] LVIS subset along
with their corresponding captions from Cap3D [23]. In the triplane fitting
stage, we have standardized the size of the 3D objects by scaling them to fit
within a unit-length bounding box. Subsequently, we placed them at the origin
and generated 100 random views for each object at a resolution of $512\times
512$.

#### Triplane fitting

We defoine the triplane size as $H=W=256,C=6$. The shared decoder used in
volume rendering is a 4-layer MLP, with each hidden layer having a width of
32. It is trained on a set of 500 objects. Subsequently, we freeze the shared
decoder and independently fit the triplane representation for each object,
implementing a linearly decayed learning rate. The balancing coefficients for
loss items are as follows:
$\lambda_{\text{L1}}=0.2,\lambda_{\text{mask}}=0.1,\lambda_{\text{depth}}=0.5,\lambda_{\text{L2}}=0.05,\lambda_{\text{TV}}=0.05,\lambda_{\text{hull}}=1.0$.
The triplanes are scaled to $[-1,1]$ to ensure a similar variance with images
for easy diffusion training later. The process of fitting one object on a
single A100 GPU takes 2 minutes.

#### Pseudo-image diffusion

We start from Stable Diffusion v2.1 and train the learnable embedding with a
learning rate of $10^{-3}$ while the remaining parameters are trained with
$10^{-4}$ for 15,000 iterations. Training is conducted using $256\times 256$
real images from LAION-5B [34] and pseudo-images with 50% probability each.
The equivalent batch size for triplanes and real images are set to 384 and
2304, respectively, to match their computational overhead. We drop the text
embedding with a 5% probability, jointly training an unconditional diffusion
model to enable classifier-free guidance [14]. Utilizing the v-objective [33]
and Min-SNR strategy [12] enhances convergence during diffusion training. The
entire diffusion training process spans about 36 hours, utilizing 8 A100 GPUs.

#### Text-to-3D generation

In pseudo-image sampling, we use DDPM [15] scheduler for 50 sample steps with
a classifier free guidance [14] scale of 5.0. It takes 3 seconds to sample a
triplane with the $256\times 256$ resolution. We implement the subsequent
refinement based on threestudio [10], utilizing DeepFloyd-IF [1] as the 2D
guidance model. The triplane-based NeRF is optimized for 2000 steps with a
batch size of 1, requiring 3 minutes on an A100 GPU. The timestep $t$ is
sampled in $\mathcal{U}(0.1,0.5)$ to introduce small corrections and prevent
excessive changes. The classifier-free guidance adopted in SDS is 20.

Figure 5: Examples generated by Dreamfusion-IF [27, 10] and our method. PI3D
can generate 3D consistent models much faster.

###  5.2 Comparisons with Other Text-to-3D Methods

We compare PI3D with general text-to-3D diffusion models, Point·E [25] and
Shap·E [19] and SDS-based method DreamFusion [27]. To evaluate the fidelity,
we adhere to the metrics used in prior works [19, 27], namely CLIP R-Precision
and CLIP similarity score [13]. CLIP R-precision measures the accuracy of
retrieving text prompts using rendered images based on CLIP similarity. We
select 4 views at different camera poses with azimuth angles $0,90,180,270$
degrees and the same elevation angles $15$ degrees, performing multiple
retrievals to obtain an average. We compare these metrics on the 415 prompts
from the DreamFusion gallery.

We first compare the pseudo-image diffusion model with the baseline text-to-3D
diffusion models. The quantitative results are reported in Tab. 1. Our model
significantly outperforms the other two methods in all the metrics, revealing
that it could generate more text-aligned 3D models. We also demonstrate the
generation quality through some qualitative cases (see Fig. 4), each presented
with 3 rendered views. The baselines, Point·E [25] and Shap·E [19], can
generate simple objects that are part of complex prompts but struggle to
capture the semantics of intricate text prompts, for example, depicting ”corn
fields” without the tractor or a ”penguin” without the skiing activity. We
believe the limited generalization ability is partly because they are both
trained only on 3D datasets with variable quality of text annotation and 3D
geometry. However, leveraging the similarity between pseudo-images and 2D real
images, our method can learn rich concepts from limited 3D data and achieve
reasonable generation quality even with prompts involving concept combination.

We also compare the refinement stage with DreamFusion [27] on the generation
efficiency and quality. We present the implementation for DreamFusion-IF in
threestudio [10]. We compare their efficiency and quality in Tab. 2 and Fig.
5, respectively. The baseline method requires tons of iterations in hours
while still performing poorly in 3D consistency and convergence. Our method
can generate 3D consistent models and converge much faster with an even
smaller batch size in only 3 minutes by leveraging a good 3D initialization
sampled from the diffusion model.

Table 2: Comparisons of the computation costs of DreamFusion-IF [27] and PI3D. Time costs are evaluated on a single A100 GPU. We achieve a 30$\times$ speedup.  Method | Iterations | Time | Batch size
---|---|---|---
DreamFusion-IF [27] | 15000 |  $\sim$ 1.5 hr | 4
Ours | 2000 | 3 min | 1
Table 3: Quantitative results of the ablation experiments on probability for training with 2D data and classifier free guidance. CFG | 2D% | ViT-B-32 | ViT-L-14
---|---|---|---
CLIP Score | R-Precision@1 | R-Precision@10 | CLIP Score | R-Precision@1 | R-Precision@10
1.0 | 0.5 | 59.2 | 8.4 | 26.9 | 45.6 | 10.1 | 28.0
3.0 | 0.5 | 65.9 | 24.6 | 55.1 | 53.2 | 29.1 | 61.7
5.0 | 0.5 | 65.9 | 25.2 | 56.7 | 53.8 | 29.3 | 63.6
7.5 | 0.5 | 66.0 | 24.4 | 56.7 | 53.8 | 29.3 | 63
10.0 | 0.5 | 64.4 | 23.4 | 54.2 | 53.2 | 28.1 | 61.6
5.0 | 0 | 61.5 | 15.7 | 41.8 | 47.4 | 16.7 | 43.1
5.0 | 0.3 | 64.7 | 22.2 | 53.9 | 51.5 | 23.9 | 57.3
5.0 | 0.7 | 65.4 | 25.2 | 56.3 | 53.4 | 30.8 | 64.0

\begin{overpic}[scale={0.4}]{figure/ablate.pdf}{ \put(20.0,180.0){\large CFG =
1.0} \put(140.0,180.0){\large CFG = 15.0} \put(260.0,180.0){\large 2D\%=0.0}
\put(370.0,180.0){\large 2D\%=0.3} \put(490.0,180.0){\large 2D\%=0.7}
\put(610.0,180.0){\large{Ours}} } \end{overpic}

Figure 6: Qualitative results of ablation experiments on different classifier-
free guidance scales and the probabilities of training with real images.

###  5.3 Ablation study

#### Depth loss in triplane fitting

To enforce a correct underlying geometry in the triplane fitting stage, we
incorporate a depth loss on the expected depth in volume rendering. We compare
the outcomes of using depth loss versus fitting without it, as illustrated in
Fig. 7. It is evident that the imposition of the depth loss constraint is
advantageous for forming the correct geometry surface. In the absence of depth
loss, the object’s surface tends to exhibit cavities near the spherical area,
resulting in hollowness in the corresponding part of the triplane. Therefore,
adopting depth loss during the fitting stage is crucial to ensure a plausible
triplane distribution. Both examples utilize the same shared decoder and
hyper-parameters, except for $\lambda_{\text{depth}}$.

\begin{overpic}[scale={0.4}]{figure/ablation_depth.pdf} \put(0.0,100.0){w/o
$\mathcal{L}_{\text{depth}}$} \put(0.0,30.0){w/ $\mathcal{L}_{\text{depth}}$}
\end{overpic}

Figure 7: Fitting the object w/ and w/o the depth loss. Rendered RGB images,
opacity images, and pseudo-images are shown.

#### Probability of training with real images

In the pseudo-image diffusion training phase, we observe that jointly fine-
tuning the model with both pseudo-images and real images is effective in
preserving the prior knowledge from the 2D pre-trained diffusion model.
Consequently, the generalization ability can be maintained to some degree.
However, using different probabilities to train with real images can lead to
distinct outcomes, such as an excessive focus on 2D image generation,
neglecting triplane quality, or a susceptibility to overfitting to 3D
datasets. We evaluate various training settings using the metrics and prompt
datasets used above. The results are presented in Tab. 3, where the
probability of utilizing real images is marked as 2D%. When training with only
3D data, the model behaves similarly to Shap·E [19]. They both lose the rich
semantic knowledge in the 2D data and are prone to overfitting to low-quality
3D data, resulting in poor performance on more generalized complex prompts.
The generalization ability of the model increases significantly as the
proportion of 2D data increases, and models trained with real images at
probabilities of 50% and 70% perform comparably. We select 50% as our final
model. Additionally, we show several qualitative results in Fig. 6. Training
without real images makes the model less generalizable and unable to
comprehend complex text inputs.

#### Classifier free guidance

We experiment with various classifier-free guidance scales to identify the
optimal match between the generated 3D content and the input text prompt. The
qualitative results for different scales are presented in Tab. 3. It is
evident that PI3D performs best when the scale is in the range of $5.0\sim
7.5$. We attribute this to the parallel training with real images, aligning
the optimal range with sampling the image diffusion model. The qualitative
results in Fig. 6 indicate that the 3D model poorly conforms to the text and
exhibits rough geometry when the scale is too small, while a large scale
introduces noisy geometry although the semantics are more text-aligned.

##  6 Limitations

Although PI3D demonstrates the ability to efficiently generate 3D consistent
shapes from text prompts, there still exist some limitations left for the
future to address. First, our training procedure involves a per-object
triplane fitting phase, which leads to an exponential increase in training
costs when scaling up to larger 3D datasets. This might be solved by encoding
triplanes from point clouds or multi-view images with an autoencoder
architecture like [25, 11]. Additionally, the capacity of our scene
representation is limited by the feature dimensions. When it comes to more
detailed 3D generation, higher dimensional features are inevitably required,
which raises the number of pseudo-images and poses a challenge in modeling the
relationships between them. This may be solved by designing more explicit and
effective feature interactions.

##  7 Conclusion

In this paper, we introduce PI3D, a novel and efficient framework that
utilizes the pre-trained text-to-image diffusion models to generate high-
quality 3D shapes in minutes. On the one hand, it fine-tunes a pre-trained 2D
diffusion model into a 3D diffusion model, enabling both 3D generative
capabilities and generalization derived from the 2D model. On the other, it
utilizes score distillation sampling of 2D diffusion models to quickly improve
the quality of the sampled 3D shapes. PI3D enables the migration of knowledge
from image to triplane generation by treating it as a set of pseudo-images. We
adapt the modules in the pre-training model to enable hybrid training using
pseudo and real images, which has proved to be a well-established strategy for
improving generalizability. The efficiency of PI3D is highlighted by its
ability to sample diverse 3D models in seconds and refine them in minutes. The
experimental results confirm the advantages of PI3D over existing methods
based on either 3D diffusion models or lifting 2D diffusion models in terms of
fast generation of 3D consistent and high-quality models. The proposed PI3D
stands as a promising advancement in the field of text-to-3D generation, and
we hope it will inspire more research into 3D generation leveraging the
knowledge in both 2D and 3D data.

## References

  * AI [2023] Stablility AI.  Deepfloyd-if.  https://github.com/deep-floyd/IF, 2023.
  * Armandpour et al. [2023] Mohammadreza Armandpour, Ali Sadeghian, Huangjie Zheng, Amir Sadeghian, and Mingyuan Zhou.  Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond.  _CoRR_ , abs/2304.04968, 2023.
  * Cao et al. [2023] Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu.  Large-vocabulary 3d diffusion model with transformer.  _CoRR_ , abs/2309.07920, 2023.
  * Chan et al. [2022] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3d generative adversarial networks.  In _CVPR_ , pages 16102–16112. IEEE, 2022.
  * Chang et al. [2015] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  _CoRR_ , abs/1512.03012, 2015.
  * Chen et al. [2023] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  _CoRR_ , abs/2303.13873, 2023.
  * Cheng et al. [2023] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G. Schwing, and Liangyan Gui.  Sdfusion: Multimodal 3d shape completion, reconstruction, and generation.  In _CVPR_ , pages 4456–4465. IEEE, 2023.
  * Deitke et al. [2023a] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi.  Objaverse-xl: A universe of 10m+ 3d objects.  _CoRR_ , abs/2307.05663, 2023a.
  * Deitke et al. [2023b] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In _CVPR_ , pages 13142–13153. IEEE, 2023b.
  * Guo et al. [2023] Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang.  threestudio: A unified framework for 3d content generation.  https://github.com/threestudio-project/threestudio, 2023.
  * Gupta et al. [2023] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz.  3dgen: Triplane latent diffusion for textured mesh generation.  _CoRR_ , abs/2303.05371, 2023.
  * Hang et al. [2023] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo.  Efficient diffusion training via min-snr weighting strategy.  _CoRR_ , abs/2303.09556, 2023.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In _EMNLP (1)_ , pages 7514–7528. Association for Computational Linguistics, 2021.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _CoRR_ , abs/2207.12598, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Ho et al. [2022a] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans.  Imagen video: High definition video generation with diffusion models.  _CoRR_ , abs/2210.02303, 2022a.
  * Ho et al. [2022b] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.  Video diffusion models.  In _NeurIPS_ , 2022b.
  * Hui et al. [2022] Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu.  Neural wavelet-domain diffusion for 3d shape generation.  In _SIGGRAPH Asia_ , pages 24:1–24:9. ACM, 2022.
  * Jun and Nichol [2023] Heewoo Jun and Alex Nichol.  Shap-e: Generating conditional 3d implicit functions.  _CoRR_ , abs/2305.02463, 2023.
  * Kajiya and Herzen [1984] James T. Kajiya and Brian Von Herzen.  Ray tracing volume densities.  In _SIGGRAPH_ , pages 165–174. ACM, 1984.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Trans. Graph._ , 42(4):139:1–139:14, 2023.
  * Liu et al. [2023] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  _CoRR_ , abs/2303.11328, 2023.
  * Luo et al. [2023] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson.  Scalable 3d captioning with pretrained models.  _CoRR_ , abs/2306.07279, 2023.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV (1)_ , pages 405–421. Springer, 2020.
  * Nichol et al. [2022] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen.  Point-e: A system for generating 3d point clouds from complex prompts.  _CoRR_ , abs/2212.08751, 2022.
  * Peng et al. [2020] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 523–540. Springer, 2020.
  * Poole et al. [2023] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In _ICLR_. OpenReview.net, 2023.
  * Qian et al. [2023] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  _CoRR_ , abs/2306.17843, 2023.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , pages 10674–10685. IEEE, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _ICLR_. OpenReview.net, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: an open large-scale dataset for training next generation image-text models.  In _NeurIPS_ , 2022.
  * Shen et al. [2021] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  _NeurIPS_ , 34:6087–6101, 2021.
  * Shi et al. [2023] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  _CoRR_ , abs/2308.16512, 2023.
  * Shue et al. [2023] J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein.  3d neural field generation using triplane diffusion.  In _CVPR_ , pages 20875–20886. IEEE, 2023.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , pages 2256–2265. JMLR.org, 2015.
  * Song et al. [2021] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _ICLR_. OpenReview.net, 2021.
  * Wang et al. [2023] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In _CVPR_ , pages 12619–12629. IEEE, 2023.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021.
  * Wu et al. [2023a] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7623–7633, 2023a.
  * Wu et al. [2023b] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, and Ziwei Liu.  Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation.  In _CVPR_ , pages 803–814. IEEE, 2023b.

\begin{overpic}[scale={0.2}]{figure/supp1.png} \put(70.0,310.0){{a beagle in a
detective's outfit}} \put(0.0,0.0){{a beautiful dress made out of garbage
bags, on a mannequin}} \put(310.0,310.0){{a car made out of sushi }}
\put(270.0,0.0){{a dachshund dressed up in a hotdog costume}}
\par\end{overpic} Figure 8: We provide more generated results with diversity.
We show 3 different outcomes for each text prompt.
\begin{overpic}[scale={0.2}]{figure/supp2.png} \put(330.0,270.0){{a peacock on
a surfboard}} \put(100.0,10.0){{an origami motorcycle}} \put(330.0,10.0){{an
amigurumi motorcycle}} \put(100.0,270.0){{a classic Packard car}}
\end{overpic}